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FLEXIBLE  METHOD  FOR  DEVELOPING  TACTICS,  TECHNIQUES,  AND  PROCEDURES 
FOR  FUTURE  CAPABILITIES 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  is  transforming  to  a  highly  mobile  Future  Force  empowered  by  the  Future 
Combat  Systems  (FCS).  As  the  FCS  capabilities  emerge,  leaders  and  Soldiers  will  need  new 
tactics,  techniques,  and  procedures  (TTP)  to  exploit  the  information-based  technologies  that 
characterize  FCS.  This  necessitates  TTP  exploration  and  development  prior  to  systems 
production.  Traditional  methods  for  developing  TTP  cannot  readily  meet  this  challenge.  New 
TTP  development  tools  must  flexibly  structure  activities  that  handle  a  wide  range  of  concepts 
and  rapidly  produce  futuristic  how-to-fight  constructs.  The  goal  of  this  research  effort  was  to 
create  an  exportable  TTP  development  method  for  use  by  Future  Force  and  FCS  developers. 

Procedure: 

Our  research  team  built  on  an  existing  cognitive  task  analysis  method  (Shadrick,  Lussier, 
&  Hinkle,  2005)  to  create  a  future-focused  TTP  development  method  called  Flexible  Method  of 
Cognitive  Task  Analysis  (FLEX).  We  developed  simulation-driven  vignettes  focusing  on  the 
use  of  FCS-related  Unmanned  Aircraft  Systems  (UAS).  The  team  crafted  knowledge  elicitation 
(KE)  tools  and  integrated  them  with  the  vignette  materials  into  a  comprehensive  TTP 
development  support  package.  We  then  conducted  trial  implementations  with  multiple  groups  of 
Soldiers  applying  the  KE  procedures  to  develop  TTP  for  employing  UAS.  Each  set  of  TTP  was 
iteratively  reviewed  and  revised  by  other  groups  of  Soldiers.  Additional  Soldiers  and  subject 
matter  experts  (SME)  integrated  multiple  sets  of  TTP  to  produce  a  final  set.  Measures  of 
effectiveness  along  with  Soldier  feedback  were  gathered  throughout. 

Findings: 

The  KE  process  produced  user-accepted  TTP  across  all  simulation  vignettes.  The 
participants  generated  more  tactics  and  techniques  than  procedures.  They  also  generated  more 
mission-related  TTP  than  other  types  (e.g.,  enemy-related).  Increasing  the  number  of 
development  sessions  increased  the  number  of  TTP  generated,  although  with  diminishing  returns 
after  around  three  sessions.  An  examination  of  the  final  set  of  TTP  compiled  by  research  team 
SMEs  revealed  that  a  majority  of  the  effective  vetting  occurred  during  the  first  two  single-source 
vetting  sessions  and  the  multiple-source  vetting  sessions.  Taken  together,  these  findings  suggest 
that  two  to  three  development  sessions,  plus  two  single-source  and  one  multiple-source  vetting 
sessions,  are  sufficient  to  efficiently  produce  effective  TTP. 

The  effectiveness  of  the  KE  method  appeared  to  be  relatively  strong.  A  majority  of  participants 
and  researchers  rated  all  aspects  of  the  KE  method  highly,  except  for  technical  aspects  of  the 
simulation.  The  low  ratings  for  the  technical  aspects  of  the  simulation  were  likely  due  to  the  use 
of  an  immature  and  procedurally  complex  simulation  system.  The  data  collection  yielded 
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lessons  learned  and  recommendations  for  improving  the  KE  method  and  selecting  simulation 
systems. 

Utilization  and  Dissemination  of  Findings: 

The  FLEX  method  for  TTP  development  supports  the  systematic  implementation  of  a 
structured,  interview-based,  and  simulation-driven  approach  that  elicits  and  refines  warfighting 
ideas  from  Soldiers.  The  products  and  findings  of  this  research  can  benefit  experts  in  the  U.S. 
Army  Training  and  Doctrine  Command,  especially  the  Future  Force  Integration  Directorate 
(FFID),  the  Army  Evaluation  Task  Force  (AETF),  and  other  agencies  working  to  develop 
concepts  and  TTP  for  the  Future  Force.  Those  who  are  specifying  and  designing  Future  Force 
capabilities  will  be  able  to  utilize  the  FLEX  TTP  method  in  their  own  TTP  development  efforts. 
Through  their  involvement  in  this  effort,  FFID  and  AETF  have  applied  and  will  continue  to  help 
ARI  refine  the  FLEX  TTP  method. 
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FLEXIBLE  METHOD  FOR  DEVELOPING  TACTICS,  TECHNIQUES, 

AND  PROCEDURES  FOR  FUTURE  CAPABILITIES 

INTRODUCTION 

With  the  U.S.  Army’s  rollout  of  the  Future  Force  and  Future  Combat  Systems  (FCS)  over 
the  next  several  years,  radically  new  technologies  will  be  introduced  to  the  field  (Welch,  2003). 
Using  an  integrated  system  of  systems  approach,  the  goal  is  to  provide  tomorrow’s  Soldiers  with 
high  battlefield  mobility  and  unprecedented  reconnaissance  and  surveillance  capabilities. 

The  FCS  family  of  systems  will  provide  unprecedented  capabilities  for  Soldiers  in  the 
U.S.  Army.  However,  Wass  de  Czege  and  Biever  (1998)  state,  “Combat  power  is  not  the  sum  of 
machine  performance;  it  requires  individual  and  organizational  competence  and  synergy”  (p.  19). 
Shadrick,  Lussier,  and  Hinkle  (2005)  also  state  that,  “A  change  in  technology  creates 
corresponding  changes  in  the  operational  and  cognitive  systems  -  resulting  in  the  transformation 
of  existing  roles,  processes,  and  procedures  and  the  development  of  new  ones”  (p.  1). 

To  maximize  the  effectiveness  of  the  planned  FCS  capabilities,  new  approaches  are 
required  to  develop  warfighting  guidance  for  integrating  and  employing  the  new  technologies. 
When  FCS  Spin  Outs  are  fielded  with  untested  operational  communication  patterns,  capabilities, 
and  weaknesses,  Soldiers  must  figure  out  the  integration  process  by  testing  it  in  the  field.  By 
developing  initial  tactics,  techniques,  and  procedures  (TTP)  before  equipment  capabilities  are 
actually  produced,  a  baseline  of  infonnation  on  changes  to  or  development  of  new  TTP  can  be 
developed  and  tested.  However,  the  Army  needs  an  innovative  method — one  that  is  structured, 
flexible,  and  measurable — to  support  rapid  development  of  initial  TTP. 

Traditional  methods  for  developing  TTP  rely  on  the  exploration  and  testing  of  new 
doctrine  and  training.  Yet  these  approaches  are  often  less  effective  than  desired  when  real  world 
implementation  is  impossible.  To  address  this  problem  and  help  ensure  effective  Spin  Out  of 
FCS  technologies,  TTP  development  methods  that  provide  structured  activities  to  measure, 
assess,  and  guide  the  process  are  needed.  Shadrick,  Lussier,  and  Hinkle’s  (2005)  flexible  method 
of  cognitive  task  analysis  (FLEX)  provides  a  promising  approach  for  TTP  development.  The 
FLEX  method  is  an  iterative,  interview-based,  and  vignette-driven  approach  that  provides  a 
structured  process  for  developing  future  concepts.  The  goal  of  this  research  and  development 
effort  was  to  expand,  implement,  and  document  the  FLEX  method  as  a  TTP  development  tool 
for  future  systems. 

This  report  describes  the  methodology  developed  to  support  the  establishment  of  future- 
focused  TTP  and  the  results  of  a  trial  implementation  of  the  methodology.  It  is  intended  to  help 
FCS  developers  as  they  work  to  make  the  Army’s  Future  Force  vision  a  reality. 

Background 

Shadrick,  Lussier,  and  Hinkle  (2005)  have  identified  two  primary  methods  of  concept 
development  traditionally  used  by  the  U.S.  Army.  The  first  method  is  an  expert  analysis  of  new 
concepts  and  technologies,  which  often  provides  general  information  about  the  integration  and 
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impact  of  future  systems.  The  second  method  involves  developing  a  replica  of  a  new  system  and 
testing  it  through  either  a  unit  exercise  or  a  simulation. 

Both  of  these  methods  pose  disadvantages.  With  the  first  method,  the  resulting  analysis 
is  typically  quite  general  in  scope  and  not  aimed  at  developing  TTP.  It  is  also  limited  in  that  it 
represents  the  views  of  one  or  few  experts.  Consequently,  the  resulting  analysis  does  not 
undergo  testing  through  implementation.  The  second  method  offers  the  advantage  of  actual 
implementation,  but  it  is  often  expensive  to  conduct.  Because  of  the  expense  involved,  there  are 
limited  opportunities  to  manipulate  variables  related  to  the  future  systems  and  to  iteratively 
develop  and  agree  upon  the  impact  of  such  systems.  A  method  is  needed  that  will  address  the 
disadvantages  of  these  two  systems.  Such  a  method  needs  to  provide  a  measurable  process  for 
eliciting  expert  knowledge  that  is  iteratively  developed  and  reviewed  by  a  range  of  Soldiers. 

Various  methods  for  eliciting  the  knowledge  of  experts  in  the  U.S.  Army  have  been  used 
(e.g.,  Klein,  1996),  and  most  have  focused  on  task  analysis-based  approaches.  With  traditional 
task  analysis,  the  focus  is  on  understanding  how  an  expert  or  experts  perform  a  job  as  well  as  the 
knowledge,  skills,  and  attitudes  required  to  perform  the  job  (Department  of  Defense,  2001). 

Also,  job  performance  has  a  specific  start  and  end  point  and  must  be  observable. 

The  use  of  cognitive  task  analysis  (CTA)  as  an  approach  to  eliciting  expert  knowledge 
(e.g.,  Schraagen,  Chipman,  &  Shalin,  2000)  has  been  increasing  because  the  approach  facilitates 
capturing  non-observable  behaviors  as  well  as  tacit  knowledge  and  processes.  This  is 
particularly  valuable  with  future  concepts  since  the  systems  and  technologies  have  not  yet  been 
developed.  In  using  CTA,  we  can  go  beyond  procedural  knowledge  and  the  behavioral  aspects 
of  an  individual’s  job  in  order  to  understand  the  “cognitive  map”  that  guides  his/her  work 
processes. 

The  FLEX  method  developed  by  Shadrick,  Lussier,  and  Hinkle  (2005)  employs  CTA 
methods  to  elicit  Soldiers’  expertise  via  structured  processes  involving  interactions  with 
simulation-based  vignettes.  The  vignettes  facilitate  making  decisions  about  how  to  employ 
future  equipment  in  light  of  specific  factors  related  to  mission,  enemy,  terrain/weather,  troops, 
time,  and  civilians  (METT-TC).  The  decisions  form  the  basis  for  developing  TTP  in  support  of 
future  concepts.  Resulting  TTP  are  iteratively  reviewed  and  refined  until  a  baseline  foundation 
for  the  new  systems  and  technologies  has  been  developed.  This  approach  was  the  focus  of  this 
research. 

The  FLEX  method  was  designed  to  consider  and  capture  future  capabilities  and  their 
implementation  by  Soldiers  in  the  field  who  have  practical  warfighting  experience.  It  harnesses 
knowledge  elicitation  (KE)  methodology  to  examine  how  Soldiers  would  employ  technologies  in 
different  types  of  missions.  By  working  with  Soldiers  from  different  military  occupational 
specialties  (MOS),  as  well  as  varying  years  of  service  and  types  of  field  experience,  we  can 
develop  a  rich  perspective  of  how  Future  Force  concepts  and  technologies  would  be  used, 
adapted,  and  advanced  by  expert  Soldiers. 
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Developed  as  an  iterative  interview  and  vignette-based  KE  approach,  the  FLEX  method 
is  designed  to  explore  future  concepts.  The  basic  features  of  the  method  are  outlined  below  and 
illustrated  in  Figure  1. 

1 .  Expert  participants  are  provided  with  a  potential  future  situation  and  are  asked  to 
solve  a  complex  problem  using  the  anticipated  capabilities. 

2.  Participants  are  asked  to  verbalize  their  responses  by  thinking  aloud. 

3.  Responses  from  participants  are  provided  to  subsequent  participants. 

4.  A  semi-structured  interview  is  used  to  probe  expert  knowledge  and  gain  a  deeper 
understanding  of  the  participant’s  reasoning. 

5.  Responses  are  reviewed  and  refined  by  subsequent  participants. 

6.  Finally,  a  small  group  of  experts  is  used  for  interactive  discussions  allowing  for 
consensus  building  and  validation. 

7.  The  consensus  outcomes  can  then  be  used  to  design  training  or  write  formal  TTP. 
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Figure  1.  The  FLEX  process  (taken  from  Shadrick,  Lussier,  &  Hinkle,  2005). 

Since  the  FLEX  method  is  grounded  in  CTA  approaches,  the  focus  is  on  facilitating  and 
capturing  decisions  made  by  Soldiers  as  they  interact  with  complex  problems  and  environments. 
Within  complex  systems,  there  are  often  multiple,  interconnected  problems  and  variables  (Funke, 
1991).  Since  decision  makers  often  have  to  respond  rapidly,  typically  under  the  pressure  of 
limited  resources  and  information,  it  can  be  challenging  to  identify  how  technology  impacts 
roles,  processes,  and  procedures.  By  examining  decisions  made  for  taking  a  specific  course  of 
action,  we  can  also  examine  the  decision  makers’  assumptions,  perceptions/misperceptions,  and 
their  use  and  adaptation  of  the  technologies  within  the  larger  system. 
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The  FLEX  method  is  similar  to  Map  Exercises  (MAPEX  )  and  other  tactical  exercises 
without  troops  (TEWT)  in  that  a  small  group  of  Soldiers  is  placed  in  a  tactical  scenario  and 
required  to  make  decisions.  However,  the  FLEX  method  has  many  advantages  over  TEWTs. 
First,  the  FLEX  method  provides  a  structured,  scientifically-based  approach  to  eliciting 
knowledge  during  and  after  the  scenario  that  TEWTs  generally  lack.  Indeed,  the  current  research 
has  developed  a  number  of  specific  instruments  designed  to  aid  in  this  process.  Second,  the 
iterative,  feed-forward  nature  of  the  method  allows  for  refinement  of  the  concepts,  TTP,  etc.,  that 
are  elicited.  This  does  not  occur  in  a  TEWT.  Third,  a  traditional  MAPEX  or  sand-table  exercise 
does  not  simulate  the  capabilities  of  the  systems  being  used  in  the  scenario.  Newer  fonns  of 
TEWTs  do  use  simulation  (and  are  therefore  closer  to  the  FLEX  method),  but  still  lack  the 
structured  KE  approach  and  iterative,  feed-forward  nature  of  the  FLEX  method. 

To  facilitate  decision  making  within  complex  environments,  simulations  offer  powerful 
tools  for  TTP  development.  Because  they  facilitate  the  capability  to  visualize,  interact  with,  and 
manipulate  variables  within  an  authentic  environment  (Gredler,  2004),  they  are  used  as  part  of 
the  FLEX  method  as  a  way  for  Soldiers  to  “interact”  with  future  capabilities  and  technologies. 
Since  most  Soldiers  and  even  Future  Force  experts  have  not  been  immersed  in  a  networked 
system  of  systems,  they  have  limited  understanding  of  employment  and  integration  of  these  new 
technologies.  Thus,  simulations  provide  valuable  tools  for  helping  envision  the  impact  of  such 
technologies  on  warfighting  TTP. 

In  summary,  the  design  of  the  FLEX  method  provides  a  framework  for  examining 
variables  that  impact  Soldiers’  employment  and  integration  of  new  technologies  within  current 
approaches. 


Technical  Objectives 

The  research  described  in  this  report  focused  on  creating  an  exportable  TTP  development 
method  capable  of  supporting  Future  Force  and  FCS  developers.  The  FLEX  method  served  as 
the  foundation  for  the  effort.  The  following  technical  objectives,  as  refined  during  the  execution 
of  the  project,  guided  the  research: 

•  Establish  a  vignette-  and  simulation-based  method  for  developing  TTP  for  employing 
Future  Force  capabilities; 

•  Implement  the  method  to  develop  TTP  for  employing  FCS  Spin  Out  3  capabilities  in 
concert  with  existing  capabilities; 

•  Refine  the  TTP  development  method  as  appropriate  based  on  results  of  the  initial 
implementation;  and 

•  Document  the  development,  trial  implementation,  and  refinement  of  the  TTP 
development  method. 
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METHOD 


Overview 

The  goal  of  this  project  was  to  develop,  implement,  and  examine  the  usability  of  a  new 
method  for  developing  TTP  for  FCS  Spin  Outs,  as  well  as  other  system  development  and 
acquisition  programs.  This  led  to  conducting  the  research  in  the  following  stages: 


•  Development  of  simulation  vignettes; 

•  Development  of  KE  process  and  instruments; 

•  TTP  development  and  review/refinement  during  implementation; 

•  TTP  finalization;  and 

•  Assessment  of  the  TTP  development/refinement  method. 

The  research  approach  combined  military  subject  matter  expertise,  behavioral  science 
knowledge,  CTA  expertise,  and  computer-based  simulation  expertise  to  execute  these  stages. 

We  relied  on  Future  Force  documentation  to  create  the  simulation-based  vignettes  that  reflected 
the  latest  concepts.  The  KE  process  was  developed  based  on  the  FLEX  method  (Shadrick, 
Lussier,  &  Hinkle,  2005).  Specific  KE  approaches  and  instruments  were  shaped  to  ensure  that 
KE  sessions  were  grounded  in  the  Future  Force  concepts  and  that  they  would  facilitate  both  TTP 
development  and  review/refinement.  Additionally,  the  KE  approaches  were  fashioned  to  ensure 
balanced  contributions  of  individual  Soldiers  as  well  as  small  group  interactions.  The  newly 
created  method  was  evaluated  by  examining  specific  outcomes  and  variables,  such  as  how  many 
TTP  were  produced  and  revised,  key  differences  between  those  in  various  roles,  and  the  overall 
success  factors  related  to  the  implementation  of  the  method. 

Development  of  Simulation  Vignettes 

In  accordance  with  the  proof-of-concept  framework,  the  development  of  TTP-focused 
simulation  vignettes  concentrated  on  FCS  Spin  Out  3,  a  family  of  Unmanned  Aircraft  Systems 
(UAS).  Vignettes  focused  on  the  company  echelon  and  below,  reflected  the  employment  of  Spin 
Out  3  UASs  in  combination  with  Spin  Out  1  and  2  capabilities,  and  accommodated  a  variety  of 
Soldier  backgrounds  and  qualifications. 

To  provide  an  overall  context  for  the  vignettes,  a  Road  to  War  was  created  to  set  the  stage 
for  the  current  tactical  conditions.  Descriptions  of  friendly  and  enemy  forces,  events,  timelines, 
and  images  were  provided  to  help  participants  understand  the  big  picture.  To  provide  a  range  of 
tactical  conditions,  different  types  of  missions,  units,  terrain,  enemy,  and  uncertainty  of  enemy 
knowledge  were  used  across  vignettes,  as  shown  in  Table  1.  An  additional  vignette  focusing  on 
company  security  patrol  served  as  a  practice  vignette  before  executing  the  ones  below. 


5 


Table  1 


Tactical  Parameters  Selected  for  Vignettes 


Vignette 

Mission 

Unit 

Terrain 

Enemy 

Knowledge  of 
Enemy 

1 

Cordon  &  Search 

Bravo  Co 

Wooded 

Attican  ADA  Section 

No  surprise 

2 

Raid 

Bravo  Co 

Urban 

Insurgent  Squad 

Surprise 

3 

Screen 

Bravo  Co 

Hilly 

Attican  Platoons 

Surprise 

4 

Movement  to  Contact  Bravo  Co 

Open 

Attican  Platoon 

No  surprise 

Each  vignette  was  designed  to  provide  approximately  1 5  to  20  minutes  of  interaction  for 
the  participants.  A  battalion  operation  order  (OPORD)  set  the  stage  for  the  family  of  missions. 
Accompanying  each  vignette  was  a  battalion  fragmentary  order  (FRAGO)  to  prompt  mission 
planning  and  accomplishment.  Also  provided  were  event  guides  for  exercise  controllers  and 
terrain  sketches. 

Vignettes  were  developed  using  the  Objective  Force-OneSAF  Objective  System  (OF- 
OOS)  software  platform  (Version  1.0.2).  This  platform  is  designed  to  be  a  composable,  next- 
generation  Computer  Generated  Force  (CGF)  modeling  software  that  represents  a  full  range  of 
operations,  systems,  and  control  processes  from  the  individual  combatant  and  platform  level  to 
brigade  levels.  The  version  of  OF-OOS  available  during  the  project  was  immature,  so  the 
operational  capabilities  were  fairly  limited.  Because  of  the  limited  capabilities  and  difficulty  of 
use,  simulation  operators  had  to  be  used  to  facilitate  execution  of  vignettes. 

Ensuring  the  development  of  quality  vignettes  was  critical  for  supporting  an  effective  KE 
process.  Thus,  the  research  team’s  SMEs  analyzed  TTP  development  needs  in  order  to  identify 
the  types  of  vignettes  sufficient  to  facilitate  TTP  development  for  FCS  purposes.  The  vignettes 
were  designed  to  provide  a  realistic  environment  and  the  relevant  information  to  feed  TTP 
development  and  to  ensure  the  recommended  TTP  development  process  would  be  sound.  We 
used  the  following  steps  to  develop  the  vignettes: 

•  Develop  vignette  storyboards. 

•  Review  and  revise  storyboards  based  on  SME  feedback. 

•  Verify  suitability  of  revised  storyboards. 

•  Develop  tactical  and  simulation  materials  for  each  vignette. 

•  Review  and  revise  vignette  materials  based  on  SME  feedback. 

•  Test  vignette  materials  in  simulation. 

As  an  important  step  in  developing  vignettes,  an  exemplar  vignette  was  first  drafted  and 
submitted  for  review  and  approval.  Then  the  remaining  vignettes  were  constructed  using  an 
iterative  develop-review-revise  cycle.  Each  vignette  was  reviewed  multiple  times  by  SMEs  in 
order  to  ensure  quality  and  tactical  accuracy. 
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Development  of  Knowledge  Elicitation  Process  and  Tools 

The  KE  process  and  tools  developed  for  this  project  were  based  on  the  stages  of  the 
FLEX  method  as  outlined  by  Shadrick,  Lussier,  and  Hinkle  (2005).  A  CTA  framework  served  as 
the  central  foundation  for  developing  the  KE  process.  The  primary  focus  was  on  harnessing  KE 
methods  and  developing  procedures  and  tools  that  would  facilitate  effective  TTP  development 
and  subsequent  review/refinement  in  concert  with  simulation-based  vignettes.  The  core  KE 
facilitation  techniques  included: 

•  Interaction  with  simulation-based  vignettes. 

•  Think  aloud  probes  eliciting  key  decisions  as  participants  role-played  during  vignettes. 

•  Individual  TTP  development  or  review/refinement  based  on  METT-TC  factors. 

•  Semi-structured  interview  regarding  TTP  outcomes. 

•  Group  discussion  of  overall  outcomes. 

The  overall  KE  process  entailed  multiple,  sequential  phases  as  described  in  Table  2: 
orientation,  interaction  with  simulation  vignette,  TTP  development  (or  review/refinement), 
summary  review,  and  group  debrief.  The  vignette-specific  steps  (Phases  2,  3,  and  4)  were 
iterated  until  all  four  vignettes  had  been  completed.  The  complete  sequence  of  phases  was 
designed  to  be  implemented  in  a  full  day,  to  capitalize  on  train-up  and  momentum  of  a  group  of 
participants  and  to  avoid  the  need  for  a  given  group  to  return  later.  The  multi-phase  process 
formed  a  systematic,  comprehensive  methodology  for  engaging  participants  in  tactical 
challenges,  surfacing  tacit  knowledge  and  procedures,  and  leveraging  the  participants’  expertise 
to  construct  TTP  for  employing  UAS  capabilities. 

Table  2 

Organization  of  Knowledge  Elicitation  Process  by  Phase 


Phase 

Step 

Welcome  and  introductions 

1.  Orientation 

Participants  view  multimedia  orientation  and  ask  questions 

Conduct  practice  exercise 

Facilitators  issue  FRAGO,  starting  with  the  first  vignette 

2.  Interaction  with 

Participants  conduct  abbreviated  troop  leading  procedures 

Simulation 

Participants  role-play  as  vignette  runs,  directing  simulation  operators 

Vignette 

Facilitators  deliver  think-aloud  probes  as  participants  role-play 

Facilitators  record  key  verbalizations,  decisions,  cues,  and  TTP 

Facilitators  lead  TTP  development  based  on  METT-TC  factors 

3A.  TTP 

Participants  record  TTP  elements  on  structured  worksheet 

Development 

Facilitators  record  notes  using  data  collection  package 

(Option  A) 

Facilitators  record  verbalizations  and  discussions  using  voice  recorders 
Facilitators  summarize  major  TTP  outcomes 

(Table  Continues) 
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Facilitators  lead  review  of  previously  developed  TTP  (METT-TC  focus) 
3B.  TTP  Review/  Participants  record  TTP  thoughts  and  revisions  on  structured  worksheet 
Refinement  Facilitators  record  notes  using  data  collection  package 
(Option  B)  Facilitators  record  verbalizations  and  discussions  using  voice  recorders 
Facilitators  summarize  revised  TTP 
Facilitators  lead  review  of  TTP  summary  for  the  vignette 

4.  TTP  Summary  Facilitators  record  revisions  to  TTP  summary  based  on  discussion 

Review  Facilitators  recap  new  TTP  summary 

Participants  repeat  steps  2,  3A/3B,  and  4  until  all  vignettes  are  complete 
Facilitators  lead  group  review  of  TTP  for  each  vignette 

5.  Group  Debrief  Facilitators  lead  group  discussion  of  overall  TTP  (across  vignettes) 

Facilitators  collect  all  materials  from  participants 


The  key  advantage  of  the  overall  KE  process  as  outlined  in  Table  2  stemmed  from  the  use 
of  multiple  approaches  supporting  multiple  data  outputs.  In  addition,  the  process  represented  our 
goal  of  balancing  group  KE  with  individual  KE,  simulation-driven  KE  with  discussion-based 
KE,  and  iterative  TTP  development  with  iterative  TTP  review/refinement. 

The  KE  process  steps  rely  heavily  on  note-taking  by  facilitators  and  hand-written 
constructions  by  participants  to  capture  raw  input  for  TTP  development.  In  addition,  facilitators 
record  participants’  verbalizations  and  discussions  using  digital  voice  recorders.  The  voice 
recordings  supplement  the  hand-written  notes  and  support  subsequent  analysis  after  the  KE 
sessions  are  completed. 

To  enable  consistent  implementation  and  iteration  of  the  KE  process,  we  developed  a 
family  of  KE  tools  in  hardcopy  format  (Table  3).  These  tools  included  an  orientation  package 
for  bringing  the  participants  quickly  up  to  speed,  planning  guides  for  facilitators,  sample  TTP  to 
serve  as  a  template  for  participants,  and  forms  to  facilitate  data  capture.  The  materials  were 
designed  to  make  a  future  facilitator’s  job  easier  while  preserving  his/her  ability  to  adapt  the 
process  and  tools  to  fit  specific  conditions  and  requirements. 

Table  3 

Tools  Directly  Supporting  the  Knowledge  Elicitation  Process 


Tool 

1 .  Participant  Orientation 

2.  Implementation  Plan 

3.  Iterative  Session  Plan 

4.  Generic  Schedule  (Daily) 


Description 

Overview  of  FCS,  simulation,  tactical  scenario,  vignettes,  and 
TTP  architecture  (PowerPoint  and  multimedia  versions) 
Description  of  resources,  schedule,  implementation  overview, 
administrative  procedures,  and  instructions  for  participants 
Graphic  roadmap  laying  out  a  candidate  sequence  of  develop¬ 
ment,  review/refinement,  and  consensus  building  sessions 
Notional  schedule  (matrix)  of  daily  administrative,  Familiarize- 

tion,  KE,  and  TTP  development/refinement  activities _ 

(Table  Continues) 
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5.  KE  Protocol 

6.  Sample  TTP 

7.  Data  Collection  Packages 


Detailed  instructions  and  materials  for  KE  facilitators  to  use  in 

planning  session  activities,  organized  by  steps 

Written  example  of  UAS-oriented  TTP,  illustrating  key  elements 

and  level  of  detail,  to  serve  as  a  template/guide 

Step-by-step  form  for  facilitators  to  use  in  capturing  knowledge 

and  procedural  data  during  KE  sessions _ 


We  integrated  the  family  of  KE  tools,  along  with  the  vignette  materials,  into  a 
comprehensive  developer's  support  package.  The  complete  package  was  intended  to  serve  as  a 
stand-alone  suite  empowering  future  TTP  developers  with  easy-to-use  job  aids. 

TTP  Development  Procedures 

When  put  into  action,  the  KE  process  with  its  implementation  materials  enables  the 
development  of  vignette-driven  TTP.  A  TTP  development  session  would  be  conducted  with  a 
small  group  of  participants  (three  to  four  Soldiers).  First,  participants  interact  with  a  simulation- 
based  vignette  while  playing  an  assigned  role  (e.g.,  Platoon  Leader  [PL]).  During  the  simulation, 
facilitators  provide  think  aloud  probes  in  order  to  elicit  each  participant’s  key  decisions  as  well 
as  factors  and  cues  related  to  why  specific  decisions  were  made.  This  occurs  during  the  course 
of  participants’  interactions  with  each  of  the  vignettes.  The  think  aloud  probes  include  questions 
such  as  “Can  you  tell  me  how  you  are  using  your  UAS?”  or  “What  factors  influenced  your 
decision  to  take  this  action?”  Facilitators  record  participants’  responses  and  audio  transcripts 
from  the  sessions  help  ensure  the  quality  of  the  data  collected. 

Once  participants  finish  role-playing  in  a  vignette,  they  complete  a  TTP  development 
worksheet  for  the  vignette.  They  individually  record  TTP  elements  specific  to  the  factors  of 
METT-TC.  After  participants  complete  the  worksheet,  the  KE  facilitator  conducts  a  semi- 
structured  interview  with  participants.  The  participants  are  interviewed  by  role,  with  the  senior 
role-player  (Company  Commander  [Co  Cdr])  being  interviewed  as  an  individual,  and  PLs  being 
interviewed  as  a  small  group. 

After  TTP  development  is  completed  for  each  simulation  vignette,  an  overall  TTP 
development  session  is  conducted  with  the  whole  group  in  order  to  craft  a  list  of  overall  TTP. 
These  TTP  serve  as  a  more  generic  set  that  potentially  applies  across  the  various  missions 
represented  in  the  vignettes.  Once  data  collection  is  complete,  individual  TTP  from  individual 
participants’  worksheets  for  each  vignette  are  compiled  into  a  list  representing  TTP  developed  by 
all  participants  in  the  day’s  group.  In  addition,  audio  transcripts  are  analyzed  to  produce 
selective  additions  to  the  group’s  vignette-specific  TTP  lists.  Also,  the  overall  list  of  TTP  is 
compiled  by  KE  facilitators. 

After  vignette-specific  and  overall  TTP  are  compiled  into  a  unified  set,  two  SMEs  with 
extensive  military  experience  review  each  TTP  and  edit  the  elements  to  clarify  meaning  and 
translate  acronyms.  The  SMEs  are  careful  to  not  change  the  meaning  of  any  TTP,  and  elements 
that  are  similar  or  repetitive  across  vignettes  remain  on  the  list  in  their  various  forms.  From  this 
process,  a  set  of  TTP  emerges  that  represents  one  specific  group’s  set  of  TTP.  Some  examples  of 
TTP  that  emerged  from  this  process  are: 
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•  Tactic:  Use  the  UAS  to  clear  lateral  routes  during  movement. 

•  Technique:  Fly  the  UAS  away  from  direct  route  of  march  whenever  possible  to  avoid 
tipping  the  enemy  as  to  your  scheme  of  maneuver. 

Additionally,  participants’  considerations  of  when  and  how  to  use  the  UAS  are  also 
documented.  An  example  of  a  consideration  is:  Using  the  UAS  in  areas  with  heavy  vegetation 
can  be  more  difficult  and  may  limit  your  ability  to  identify  the  enemy. 

Because  of  the  large  number  of  TTP  that  may  be  developed  by  each  group,  we  realized 
the  difficulty  of  reviewing  long  lists  of  TTP  for  each  vignette  without  some  understanding  of  the 
participants’  overall  strategy  for  accomplishing  the  mission  presented  in  the  simulation  vignette. 
Thus,  an  SME  with  military  expertise  reviewed  the  long  list  of  TTP  for  each  vignette  and 
developed  a  TTP  summary.  The  purpose  of  the  summary  was  to  provide  subsequent  participants 
with  a  TTP  development  group’s  overall  strategy  for  accomplishing  the  mission.  An  example  of 
such  a  summary  follows: 

TTP  Summary  for  Vignette  1:  Use  UAS  to  support  reconnaissance  during  movement 
(the  technique  is  over  the  shoulder).  Be  sure  to  deconflict  the  air  space  and  assign 
platoon  sectors.  Keep  at  least  one  UAS  in  reserve.  Use  the  Commander’s  UAS  for 
command  and  control.  Once  the  cordon  is  established,  use  the  UAS  to  identify  targets  on 
the  objective  and  to  call  for  fire.  The  UAS  should  observe  from  a  maximum  distance  in 
order  to  avoid  direct  fire  contact.  The  UAS  can  provide  early  warning  on  the  far  side  of 
the  objective  during  the  search. 

Such  a  summary  was  placed  at  the  beginning  of  the  list  of  TTP  for  each  vignette.  Also, 
emerging  TTP  as  well  as  considerations  (TTP-C)  were  kept  in  their  original  order,  with  their 
original  vignette,  and  grouped  by  factors  related  to  METT-TC. 

TTP  Revision  Procedures 

Two  types  of  TTP  review/refinement  (vetting)  sessions  can  be  conducted — single-source 
and  multiple-source  sessions.  During  a  single-source  session,  participants  receive  TTP  that 
originated  in  a  single  TTP  development  session.  During  a  multiple-source  session,  participants 
receive  cumulative  TTP  resulting  from  all  of  the  preceding  TTP  development  and  single-source 
vetting  sessions. 

Single-Source  Vetting  Sessions 

The  TTP  review/refinement  procedure  mirrors  much  of  the  same  process  as  the  TTP 
development  procedure,  except  that  participants  receive  the  outputs  from  previous  groups  to 
establish  a  starting  point  for  the  review/refinement  process.  As  participants  role-play  during 
simulation-based  vignettes,  facilitators  provide  think  aloud  probes  in  order  to  uncover  their 
decisions  as  well  as  why  the  decisions  are  made.  The  think  aloud  probes  include  questions  such 
as  “Can  you  tell  me  why  you  made  that  decision?”  and  “What  caused  you  to  choose  that  course 
of  action?”  Again,  facilitators  record  participants’  responses,  and  audio  transcripts  from  the 
sessions  later  help  ensure  the  quality  of  the  data  collected. 
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Following  the  end  of  a  simulation  vignette,  the  participants  complete  a  worksheet  asking 
them  to  rcvicw/rcline  previously  developed  TTP.  This  worksheet  gives  the  vignette-specific 
TTP  preceded  by  the  TTP  summary  developed  by  a  SME.  Participants  are  asked  to  first  review 
the  TTP  summary,  rate  it  for  acceptability,  and  edit  it  as  needed.  Then  they  are  asked  to  review 
the  TTP-C  developed  for  each  vignette  and  to  specify  whether  they  should  be  kept,  modified,  or 
deleted.  Participants  are  asked  to  explain  why  the  specific  TTP-C: 

•  Is  good  and  should  be  kept; 

•  Should  be  modified  and  how;  or 

•  Should  be  deleted. 

Once  participants  complete  the  worksheet,  the  KE  facilitator  conducts  a  semi-structured 
interview  with  participants  to  review  each  of  their  ratings  and  to  come  to  consensus,  if  possible, 
on  each  TTP-C.  The  participants  are  interviewed  by  role,  with  the  senior  role-player  (Co  Cdr) 
being  interviewed  as  an  individual,  and  PLs  being  interviewed  as  a  small  group.  After  the  TTP 
review/refinement  process  is  completed  for  each  vignette,  participants  also  review  the  summary 
TTP  following  the  same  process. 

When  data  collection  is  complete,  results  of  each  participant’s  TTP-C  ratings  are  compiled  and 
entered  into  a  spreadsheet  noting  whether  each  TTP-C  is  to  be  kept,  modified  or  deleted. 
Additionally,  edits  made  by  participants  to  specific  TTP-C  are  compiled.  Again,  two  SMEs  with 
extensive  military  experience  review  the  revised  set  of  TTP-C  to  correct  for  unclear  meaning  or 
acronyms.  The  resulting  set  of  TTP  is  again  reviewed  and  refined  one  or  more  times  during 
subsequent  sessions  (groups),  until  each  set  of  TTP-C  reaches  a  desired  degree  of  refinement. 

Multiple-Source  Vetting  Sessions 

Multiple-source  vetting  sessions  can  be  conducted  to  integrate  cumulative  TTP  and  build 
consensus.  Because  this  approach  is  more  detailed  and  takes  more  time  to  conduct,  each  group 
of  participants  would  focus  on  just  two  vignettes  in  a  one-day  schedule. 

In  single-source  TTP  review/refinement  sessions,  participants  review  a  single  TTP 
summary  for  each  vignette  along  with  a  list  of  TTP-C  elements.  However,  in  multiple-source 
vetting  sessions,  participants  are  first  asked  to  review  as  many  TTP  summaries  for  Vignettes  1 
and  2  as  were  produced  by  the  previous  development/refinement  groups.  These  summaries  are 
edited  by  an  SME  in  order  to  ensure  clear  meaning.  The  vetting  group  then  reviews,  rates,  and 
refines  (when  needed)  each  TTP  summary.  Then  they  write  their  own  TTP  summary  by  editing 
existing  ones  or  by  drafting  their  own.  The  participants  then  work  together  to  create  one  team- 
based  TTP  summary  that  combines  the  best  elements  from  other  TTP  summaries  and  adds  new 
elements  as  deemed  important. 

In  the  next  step,  the  same  participants  may  rate  multiple  TTP-C  developed  by  different 
development  groups.  The  multiple  TTP-C  can  be  presented  again  and  grouped  by  vignette  and 
by  specific  METT-TC  factors.  Participants  rate  and  review  these  multiple  sets  of  TTP-C  for 
each  vignette  and  specify  whether  each  one  should  be  kept,  modified,  or  deleted.  Again,  they 
also  modify/refine  any  TTP-C  that  they  decide  need  refinement. 
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In  order  to  develop  a  unified  set  of  TTP,  independent  SMEs  can  integrate  the  team-based 
TTP  summaries  as  well  as  the  vignette-specific  TTP  to  create  a  final  set.  This  step  enhances  the 
quality  of  the  TTP-C  so  they  are  clear  in  meaning  and  easy  to  understand.  Thus,  SMEs  who  are 
experienced  writers  play  a  critical  role  in  producing  high  quality  TTP. 

Assessment  of  the  FLEX  Method  Implementation 

To  accomplish  an  important  technical  objective  of  the  project,  we  planned  and  conducted 
a  series  of  trial  implementations.  The  resulting  KE  sessions  applied  the  methods  and  the  KE 
tools  described  in  the  foregoing  sections,  with  Soldiers  in  the  loop.  Data  collection  efforts 
focused  on  capturing  TTP  and  documenting  the  KE  process  for  the  purpose  of  testing  the 
implementation  of  the  FLEX  method. 

Data  Collection  Instruments 

Table  4  lists  the  data  collection  instruments  prepared  for  the  trial  implementations.  These 
instruments  served  to  capture  data  with  which  to  gauge  the  effectiveness  and  efficiency  of  the 
KE  method.  The  various  instruments  were  completed  by  participants  and  research  team 
members  (serving  as  KE  facilitators  or  simulation  operators).  Some  of  the  instruments  would  be 
optional  for  future  TTP  development  efforts  (e.g.,  participant  profile,  feedback  survey,  problem 
logs). 

Table  4 

Data  Collection  Instruments  Used  During  Trial  Implementations 


Instrument 


1 .  Participant  Profile 

2.  TTP  Development  Data 
Collection  Package 

3.  TTP  Review/Refinement 
Data  Collection  Package 

4.  Worksheet:  Developing 
UAS  TTP 

5.  Worksheet:  Reviewing/ 
Refining  UAS  TTP 

6.  Participant  Feedback 
Survey 

7.  Procedural  Problems  Log 


Description 

Demographic  survey  completed  by  participants  to  self-report 
their  educational  background,  military  assignments,  and 
knowledge  of  FCS  capabilities 

Detailed  form  used  by  facilitators  during  TTP  development  to 
record  think  aloud  data  as  well  as  semi-structured  interview 
responses  related  to  UAS  usage  (see  Appendix  B) 

Detailed  form  used  by  facilitators  during  TTP  refinement  to 
record  think  aloud  data  as  well  as  semi-structured  interview 
responses  related  to  UAS  usage  (see  Appendix  C) 

Form  completed  by  participants  to  develop  new  TTP  and  to 
answer  related  questions  (see  Appendix  B) 

Fonn  used  by  participants  to  review  and  refine  previous  TTP 
and  to  answer  related  questions  (see  Appendix  C) 
Questionnaire  completed  by  participants  to  provide  feedback 
on  Flex  TTP  process  and  procedures  (see  Appendix  D) 

Fonn  used  by  facilitators  to  record  daily  procedural  problems 
(e.g.,  participant  leaving  early)  as  well  as  their  impact 

(Table  Continues) 
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8.  Technical  Problems  Log 

9.  Master  Participant  List 

10.  Transcripts  of  Audio  Files 


Form  used  by  simulation  operators  to  record  technical 
problems  (e.g.,  computer  problems)  as  well  as  their  impact 
Form  used  by  facilitators  to  log  participants  and  their 
identification  numbers  for  each  daily  session 
Transcripts  of  voice  recordings  for  each  data  collection 
session,  prepared  for  subsequent  use  by  investigators _ 


As  indicated  in  Table  4,  selected  data  collection  instruments  are  contained  in  appendixes 
of  this  report. 

Implementation  Procedures 

Data  collection  was  conducted  over  three  separate  weeks  at  two  different  sites,  with  one 
site  furnishing  participants  for  two  different  weeks.  Figure  2  shows  the  sequence  of 
development  and  review/refinement  sessions  that  we  used  to  conduct  our  research.  Each  data 
collection  period  lasted  3-5  days.  At  the  first  site,  armor  and  cavalry  leaders  (officers, 
noncommissioned  officers)  representing  platoon  and  company  echelons  participated  in  three  KE 
sessions.  At  the  second  site,  Army  Evaluation  Task  Force  (AETF)  leaders  (Week  2)  and  Future 
Force  Integration  Directorate  (FFID)  personnel  (Week  3)  participated  in  five  sessions, 
respectively.  Each  data  collection  session  lasted  6-7  hours,  with  appropriate  breaks  provided  for 
participants  throughout  the  day. 


MON 


TUE 


GRP  1 
24-26  OCT 


GRP  2 
5-9  NOV 


GRP  3 
3-7  DEC 


WED 


THU 


FRI 


Figure  2.  Research  schedule  of  TTP  development  and  review/refinement  sessions.  Circles 
indicate  TTP  development  groups  and  squares  indicate  single-source  review/refinement  groups. 
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In  the  single-source  TTP  review/refinement  sessions,  participants  reviewed  a  single  TTP 
summary  for  each  vignette  in  combination  with  a  list  of  TTP-C  for  each  vignette.  However,  in 
multiple-source  vetting  sessions,  participants  were  first  asked  to  review  five  TTP  summaries  for 
Vignettes  1  and  2  (or  Vignettes  3  and  4)  that  were  developed  by  five  former  development 
groups.  These  summaries  had  again  been  edited  by  an  SME  in  order  to  ensure  clear  meaning. 
The  summaries  from  development  sessions  one,  two,  and  three  were  previously  vetted  three, 
two,  and  one  times  respectively,  while  the  material  from  development  sessions  four  and  five  was 
not  previously  vetted.  Each  group  then  reviewed,  rated,  and  refined  (when  needed)  each  TTP 
summary.  Then  they  were  asked  to  write  their  own  TTP  summary  by  editing  existing  ones  or  by 
crafting  their  own.  The  KE  facilitator  then  asked  participants  to  work  together  to  create  one 
team-based  TTP  summary  that  would  combine  the  best  elements  from  other  TTP  summaries  and 
add  new  elements  that  were  important.  These  final  TTP  summaries  for  each  vignette  were 
recorded. 

The  trial  implementation  included  five  TTP  development  sessions,  six  single-source 
vetting  sessions,  and  finally  two  multiple-source  vetting  sessions.  Once  the  data  collection  was 
finished,  all  data  were  cataloged  and  inventoried  to  ensure  all  documentation  was  complete. 
Also,  audio  files  were  transcribed  after  each  data  collection  session.  All  data  were  grouped  and 
compiled  into  multiple  spreadsheets.  Two  members  of  our  research  team  entered  data  into 
spreadsheets,  with  random  quality  assurance  checks  performed  by  other  team  members. 

In  order  to  produce  an  integrated  set  of  TTP,  two  of  the  team’s  SMEs  independently 
reviewed  the  team-based  TTP  summaries  as  well  as  the  vignette-specific  TTP  and  combined 
them  to  create  a  final  family  of  TTP.  Because  participants  in  this  project  represented  different 
ranks,  educational  levels,  and  experience,  the  quality  of  the  raw  TTP-C  varied  greatly.  Quality 
TTP-C  need  to  be  clear  in  meaning  and  well  written.  In  fact,  some  participants  expressed 
concern  with  their  lack  of  writing  and  editing  abilities.  Thus,  the  role  of  the  research  SMEs  in 
editing  TTP-C  as  well  as  TTP  summaries  was  critical  to  producing  a  quality  set  of  TTP  for  each 
vignette  and  for  overall  purposes. 


Data  Analysis 

To  analyze  the  data,  we  used  a  mixed  methods  approach  (Creswell,  2002),  which 
incorporated  both  qualitative  and  quantitative  methods.  Quantitative  methods  were  used 
primarily  to  answer  key  research  questions,  with  qualitative  analysis  serving  as  a  secondary 
method  of  analysis.  The  number  of  items  developed/vetted  was  tabulated  according  to  session 
number,  session  type,  role  played,  METT-TC,  vignette,  and  TTP-C.  Ratings  for  TTP-C  and  KE 
procedures  were  tabulated  as  well.  Average  percentages/counts  and  standard  deviations  were 
calculated  and  used  as  the  primary  quantitative  parameters. 

RESULTS  AND  DISCUSSION 

The  effectiveness  of  the  current  KE  method  to  develop  TTP  was  assessed  through 
multiple  measures.  Each  measure  was  designed  to  address  a  separate  aspect  of  the  methodology. 
The  following  aspects  of  the  methodology  were  examined: 
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•  Participants. 

•  Implementation  Effectiveness. 

•  Echelon  Differences. 

•  Saturation  Effects. 

•  Quality  and  Origin  of  Final  TTP. 

•  Lessons  Learned. 

•  Methodology  Improvements. 

The  method  of  data  collection,  resulting  data  analysis,  and  conclusions  reached  by  our 
research  team  will  be  presented  for  each  aspect  of  the  methodology  examined. 

Participants 

A  total  of  48  participants  took  part  in  the  research  project  with  13  assigned  the  role  of  Co 
Cdr  and  35  serving  as  PLs.  Military  experience  varied  greatly  between  the  participants  in  terms 
of  rank  (see  Figure  3),  MOS  (see  Table  5),  and  length  of  service.  The  average  number  of  months 
in  service  was  126.90  (standard  deviation  [SD] =99.91),  with  the  longest  tenure  being  420 
months  and  the  shortest  14  months.  There  was  a  positive  correlation  between  length  of  service 
and  session  order,  r=.56,/?<.001,  indicating  that  more  experienced  participants  took  part  in  the 
later  KE  sessions.  The  participants  from  development  session  3  were  the  exception  to  this  trend, 
being  more  seasoned  in  terms  of  years  of  service  than  other  participants  in  the  development 
sessions.  However,  three  of  the  four  participants  in  the  third  development  session  were  supply 
specialists,  which  effectively  limited  their  knowledge  relevant  to  the  tactical  exercises.  Overall, 
77.27%  of  participants  indicated  they  had  previous  experience  developing  standing  operating 
procedures  (SOP)  or  TTP,  79.55%  reported  they  were  familiar  with  FCS  and  Spin  Outs,  but  only 
9.09%  noted  they  had  prior  experience  with  unmanned  vehicles. 
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Figure  3.  Participants’  military  rank  according  to  role  assignment. 

Implementation  Effectiveness 

Implementation  effectiveness  was  assessed  by  measuring  (a)  our  ability  to  implement  the 
methodology  and  (b)  the  perceived  effectiveness  of  the  methodology  for  eliciting  knowledge. 
These  dimensions  were  measured  by: 

•  Tabulating  significant  events  (procedural  and  technical  problems  recorded  on  daily 
logs)  across  development  and  vetting  sessions. 

•  Assessing  our  ability  to  complete  a  KE  session  according  to  the  time  allotted  by  daily 
schedule. 

•  Gauging  our  ability  to  compile  and  summarize  developed  and  vetted  TTP  to  produce 
compiled  materials  for  subsequent  vetting  sessions. 

•  Quantifying  the  researchers’  and  participants’  perceptions  of  methodology  and 
implementation  effectiveness  via  rating  techniques. 

Data  from  the  significant  event  (problem)  logs  were  grouped  according  to  five  general 
areas — simulation,  participants,  equipment,  timing,  and  materials.  For  the  purpose  of  the  project, 
a  significant  event  was  defined  as  any  event  that  occurred  during  a  session  that  deviated  from  the 
prescribed  procedure  and  could  adversely  affect  the  data.  For  instance,  significant  events 
included  simulation  workstation  crashes,  participants’  late  arrivals  or  early  departures, 
typographical  errors  or  gaps  in  materials,  and  so  forth.  Infonnation  on  the  frequency  and  type  of 
significant  events  is  provided  in  the  following  paragraphs. 
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Table  5 


Number  of  Participants  According  to  Military  Background 


MOS 

Officer/Enlisted 

Title 

Number 

11A 

Officer 

Infantry  officer 

4 

1  IB 

Enlisted 

Infantryman 

6 

12 

Officer 

Engineer  officer 

1 

13A 

Officer 

Field  Artillery  officer,  general 

2 

13F 

Enlisted 

Fire  Support  specialist 

1 

14A 

Officer 

Air  Defense  Artillery  officer 

1 

15B 

Officer 

Aviation  Combined  Arms  Ops  officer 

1 

19 

Enlisted 

Armor  CSM 

2 

19A 

Officer 

Armor  officer,  general 

11 

19B 

Officer 

Armor  officer 

2 

19C 

Officer 

Cavalry  officer 

1 

19D 

Officer 

Cavalry  Scout 

3 

19K 

Enlisted 

M 1  Armor  Crewman 

5 

19Z 

Enlisted 

Armor  senior  sergeant 

2 

21B 

Officer 

Combat  Engineer  officer 

1 

35D 

Officer 

All  Source  Intelligence  officer 

1 

68W 

Enlisted 

Health  Care 

1 

GS-12 

Civilian 

Training  Instructor 

1 

88M38 

Enlisted 

Transportation  Instructor 

1 

92Y40 

Enlisted 

Supply  specialist 

1 

Total 

48 

Simulation-Related  Events 

There  were  two  types  of  significant  events  related  to  the  simulation — single  workstation 
failures  (57.14%)  and  complete  system  failures  (42.86%).  At  least  one  significant  event 
occurred  due  to  simulation-related  factors  in  1 1  of  13  sessions  (84.62%).  Furthermore,  in  8  of 
the  13  sessions  (61.54%),  multiple  simulation-related  significant  events  occurred.  In  at  least  one 
development  session,  the  instability  of  the  simulation  likely  led  to  fewer  than  average  TTP  being 
developed.  In  fact,  during  that  session  (Development  4)  one  of  the  vignettes  was  tenninated 
early.  The  instability  of  the  system  is  most  likely  attributable  to  the  use  of  an  immature  version 
of  OF-OOS  to  run  the  simulation  and  the  demands  placed  on  the  computer  systems  by  simulation 
software.  As  future  refinements  are  made  to  the  OF-OOS  operating  suite,  the  number  of 
simulation-related  issues  should  decrease.  Alternatively,  other  platfonns  for  simulations  may  be 
utilized  in  the  future. 

Participant-Related  Events 

Participant-related  significant  events  occurred  in  9  of  13  sessions  (69.23%).  A  majority 
of  these  events  involved  participants  arriving  late  or  not  showing  up.  While  participant-related 
events  occurred  rather  frequently,  they  did  not  have  a  major  impact  aside  from  adding  some 
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variability  in  the  data  between  sessions.  We  were  able  to  complete  the  daily  sessions  by  the 
scheduled  time,  indicating  that  there  was  sufficient  flexibility  built  into  the  daily  schedule  to 
accommodate  multiple  delays. 

Equipment-Related  Events 

Equipment-related  significant  events  occurred  in  3  of  13  sessions  (23.08%).  All 
equipment-related  events  involved  the  voice  recorders,  but  they  did  not  significantly  affect  the 
data  collection  because  of  redundant  mechanisms. 

Timing-Related  Events 

Late  arrival  by  participants  was  the  primary  cause  of  timing-related  significant  events. 
While  late  participant  arrivals  translated  into  late  start  times  in  6  of  the  13  sessions  (46.16%),  the 
delays  did  not  dramatically  impact  the  daily  schedule  even  when  combined  with  delays  related  to 
the  simulation.  Each  vignette  was  completed  according  to  the  scheduled  time  allocated  (+/-  15 
minutes),  and  each  daily  session  was  completed  within  the  scheduled  time  (+/-  20  minutes).  To 
make  up  time,  the  exercise  director  occasionally  ended  an  event  (such  as  vignette  execution) 
early,  when  doing  so  would  not  materially  affect  tactical  considerations. 

Materials-Related  Events 

Significant  events  stemming  from  KE  materials  occurred  in  4  of  13  sessions  (30.77%). 
Issues  related  to  materials  tended  to  be  typographical  errors  and  occurred  during  vetting  sessions 
of  recently  compiled  and  summarized  TTP.  Future  KE  efforts  may  benefit  from  either 
streamlining  the  TTP  compilation  and  summarization  process,  or  by  providing  more  time 
between  sessions  that  require  compiling  and  summarizing  TTP. 

Effectiveness  Ratings 

Participant  and  researcher  feedback  forms  were  used  to  assess  the  perceived  effectiveness 
of  the  current  KE  method.  Both  groups  rated  aspects  of  the  KE  sessions  on  a  five-point  scale 
(strongly  disagree  to  strongly  agree)  across  several  dimensions.  As  necessary,  scores  on  the  five- 
point  scale  were  transformed  so  that  higher  scores  reflected  positive  ratings.  As  shown  in 
Figures  4  and  5,  a  majority  of  participants  and  researchers  rated  the  KE  method  positively,  with 
the  exception  of  the  simulation.  Less  than  half  the  raters  in  both  groups  gave  positive  ratings  for 
the  simulation.  It  should  be  noted  that  “neutral”  responses  are  not  represented  in  Figures  4  and 
5. 
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Figure  4.  Participant  ratings  of  effectiveness  of  various  methodology  components. 
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Figure  5.  Researcher  ratings  of  effectiveness  of  various  methodology  components. 
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The  ability  of  the  current  KE  methodology  to  facilitate  the  production  of  TTP  was 
assessed  by  examining  the  distribution  of  TTP  elements  along  four  dimensions:  vignettes  (1 
through  4),  type  of  element  (tactic,  technique,  procedure,  or  consideration),  development  session 
(1  through  5),  and  METT-TC  factors  (mission,  enemy,  terrain  and  weather,  troops,  time,  and 
civilians).  The  relative  frequencies  of  elements  for  each  dimension  are  presented  in  Table  6. 

The  influence  of  the  four  dimensions  will  be  addressed  in  the  following  paragraphs. 

Table  6 

Multi-Dimensional  Breakout  of  TTP  Items  Generated  During  Development  Sessions 


Vignette: 

One 

Two 

Three 

Four 

Average  (%) 

25.00 

31.00 

27.00 

17.80 

SD 

9.82 

9.00 

9.00 

8.14 

TTP-C: 

Tactic 

Technique 

Procedure 

Consideration 

Average  (%) 

48.00 

35.60 

3.40 

14.20 

SD 

16.84 

12.90 

4.16 

7.46 

Session: 

One 

Two 

Three 

Four 

Average  (number) 

35.00 

28.25 

20.75 

16.50 

SD 

10.00 

10.27 

7.89 

5.56 

METT-TC: 

Mission 

Enemy 

Terrain 

Troops 

Average  (%) 

52.00 

14.40 

14.20 

9.60 

SD 

16.66 

1.52 

5.63 

5.50 

Time 

8.40 

4.56 


Civilians 

2.20 

1.64 


Vignettes 

Productivity  of  the  development  sessions  was  consistent  across  vignettes  with  the 
exception  of  a  lower  number  of  TTP  items  in  Vignette  4.  Since  Vignette  4  was  always  the  final 
vignette  of  the  day,  we  cannot  determine  if  vignette  order  or  the  content  of  Vignette  4  accounted 
for  the  reduced  productivity.  We  considered  varying  the  vignette  order  as  part  of  the  research 
design  but  decided  against  it  to  maintain  continuity  in  the  tactical  storyline  across  vignettes. 
Individuals  conducting  KE  sessions  in  the  future  may  wish  to  vary  vignette  order. 

TTP-C  Components 

Participants  generated  substantially  more  tactics  and  techniques  than  procedures  or 
considerations.  The  difference  may  result  from  the  facilitators  instructing  the  participants  to 
focus  on  developing  tactics  and  techniques  because  of  the  immature  stage  of  the  UAS.  The  exact 
specifications  and  abilities  of  the  Spin  Out  UAS  have  yet  to  be  detennined.  Therefore,  we  felt  it 
would  not  be  fruitful  to  develop  specific  procedures  for  a  UAS  without  knowing  what  it  was 
capable  of  performing. 

Development  Sessions 
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The  number  of  generated  TTP  items  fluctuated  across  the  five  development  sessions. 
While  the  fourth  development  session  produced  the  fewest  items,  the  difference  is  most  likely 
due  to  the  higher  number  of  simulation-related  problems  that  occurred  during  that  session.  The 
third  session  produced  fewer  items  most  likely  due  to  three  of  the  four  participants  having  their 
military  backgrounds  in  supply  operations.  The  remaining  variability  between  development 
sessions  is  most  likely  attributable  to  error  variance  produced  by  (a)  participants  with  different 
levels  of  ability,  military  experience,  and  motivation,  (b)  number  and  type  of  significant  events, 
and  (c)  other  sources  of  natural  variability. 

METT-TC  Factors 

Participants  generated  more  items  related  to  “mission”  than  any  other  METT-TC 
element,  while  they  generated  the  fewest  items  for  the  “civilian”  element.  The  dearth  of  civilian- 
related  TTP  items  may  result  from  the  simulation  capabilities.  The  limitations  of  the  OF-OOS 
version  precluded  the  effective  portrayal  of  civilian  elements  in  the  vignettes.  Since  civilians 
were  absent  in  the  simulations,  they  were  often  not  factored  into  participants’  TTP. 

Echelon  Differences 

Researchers  assigned  the  participants  to  serve  as  a  Co  Cdr  or  a  PL,  based  in  part  on  a 
review  of  participant  demographic  information.  The  role  of  Co  Cdr  was  generally  assigned  to 
the  participant  with  the  greatest  level  of  expertise,  typically  based  on  highest  rank  or  longest 
relevant  military  experience.  The  perfonnance  at  each  echelon  was  recorded  separately  across 
the  development  and  vetting  sessions  to  examine  the  effects  of  role  assignment  on  TTP 
development  and  vetting.  Although  each  development  and  vetting  session  included  multiple 
PLs,  the  performance  of  each  PL  was  recorded  individually. 

Development  of  TTP 

The  effect  of  the  role  assignment  (PL  vs.  Co  Cdr)  on  TTP  development  was  assessed  by 
calculating  the  average  number  of  TTP  generated  per  participant  according  to  echelon  across 
development  sessions  and  vignettes.  In  addition,  the  TTP  generated  were  analyzed  according  to 
the  level  of  focus  (i.e.,  task-specific  TTP  related  to  the  “close  fight”  vs.  higher  level  command 
and  control  [C2]  issues). 

Echelon.  Participants  in  the  Co  Cdr  role  generated  more  TTP  per  development  session 
on  average  (M=33.80,  SD=  9.76)  than  participants  in  the  PL  role  (M=25.47,  ,SD=13.01).  As  seen 
in  Figure  6,  the  quantitative  difference  in  favor  of  Co  Cdrs  held  up  across  vignettes.  That  the 
magnitude  of  the  difference  varied  somewhat  between  vignettes  probably  reflects  measurement 
error. 
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Figure  6.  Average  number  (and  SD )  of  items  generated  per  vignette  by  participant  role. 

Level  of  focus.  Participants  in  the  Co  Cdr’s  role  generated  more  TTP  that  focused  on  C2 
issues  and  overall  performance  of  the  group.  That  is,  Co  Cdrs  tended  to  utilize  the  UAS  to 
support  mission  planning,  coordination  between  subordinate  units,  and  asset  support  and 
allocation.  Conversely,  PLs  generated  TTP  that  were  more  task-specific,  focused  on  specifying 
where,  when,  and  how  the  UAS  should  be  flown.  For  example,  PLs  commented  more  frequently 
on  issues  such  as  using  the  UAS  to  reconnoiter  a  specific  area  and  the  flight  patterns  related  to 
the  reconnaissance  (see  Table  7). 

Table  7 

Average  Percent  of  TTP  Generated  by  Role  and  Focus  (Task  vs.  C2) 


Session 

Role 

Focus 

One 

Two 

Three 

Four 

Five 

Average 

PL 

Task 

81.25 

77.46 

83.64 

82.61 

86.36 

82.26 

C2 

18.75 

22.54 

16.36 

17.39 

13.64 

17.74 

Co  Cdr 

Task 

65.12 

64.29 

64.29 

65.00 

61.11 

63.96 

C2 

34.88 

35.71 

35.71 

35.00 

38.89 

36.04 
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Conclusion.  Participants  working  in  the  Co  Cdr’s  role  were  generally  more  productive 
during  development  sessions,  on  average  generating  more  TTP.  Further,  the  Co  Cdr  role  players 
tended  to  generate  TTP  with  the  broader  mission  in  mind.  Since  we  deliberately  tried  to  assign 
participants  with  the  most  military  experience  to  the  Co  Cdr’s  role,  we  cannot  clearly  determine 
if  the  higher  productivity  and  broader  level  of  focus  were  byproducts  of  those  participants  being 
placed  in  the  Co  Cdr's  role,  or  byproducts  of  their  greater  military  experience. 

Vetting  of  TTP 

Single-source  vetting.  The  effect  of  role  assignment,  PL  vs.  Co  Cdr,  on  vetting  TTP  was 
assessed  by  computing  the  average  percent  of  TTP  items  deleted  or  refined  during  single-source 
vetting  sessions.  As  seen  in  Figure  7,  participants  in  the  Co  Cdr’s  role  were  generally  more 
active  during  vetting  sessions,  deleting  or  refining  more  TTP  on  average  (M=21.02,  577=14.2 1 ) 
than  participants  in  the  PL  role  (M=16.71,  577=6.48).  In  four  of  the  six  vetting  sessions  Co  Cdrs 
deleted  or  refined  more  TTP  items  than  participants  in  the  PL  role.  In  two  sessions  the  PLs 
vetted  more  TTP  on  average  than  the  Co  Cdrs.  In  one  of  these  sessions,  the  participant  in  the  Co 
Cdr  role  left  the  session  early  due  to  a  Family  emergency.  In  the  other  session  the  participant 
assigned  as  Co  Cdr  had  the  highest  military  rank,  but  two  fellow  participants  had  comparable  or 
greater  military  experience  (Co  Cdr  144  months,  PLs  152  and  144  months,  respectively).  Thus, 
the  two  instances  when  PLs  vetted  more  TTP  items  than  Co  Cdrs  can  be  explained  by  a 
participant-related  significant  event  and  a  participant  variable,  respectively. 
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Figure  7.  Average  percent  (and  577)  of  TTP  modified  or  deleted  during  single-source  vetting 
sessions,  by  participant  role. 
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The  total  number  of  items  modified  or  deleted  varied  across  vetting  sessions  (see  Figure 
7).  Vetting  sessions  one,  two,  and  three  indicate  first  round  vettings  for  TTP  produced  in 
development  sessions  one,  two,  and  three  respectively.  Vetting  sessions  four  and  five  were 
second  round  vettings  for  development  sessions  one  and  two  respectively,  while  the  sixth  vetting 
session  was  the  third  round  of  vetting  conducted  on  TTP  from  development  session  one.  The 
higher  numbers  of  items  vetted  during  the  second  and  fifth  vetting  sessions  most  likely  reflect  a 
greater  need  to  refine  TTP  produced  during  the  second  development  session. 

Multiple-source  vetting.  The  analysis  of  TTP  vetting  as  a  function  of  assigned  role  was 
extended  to  multiple-source  vetting  sessions.  Comparison  across  vetting  sessions  was  not 
practical  since  only  two  multiple-source  vetting  sessions  were  conducted,  during  which  each 
group  received  only  a  partial  complement  of  TTP  (Vignettes  1  and  2,  or  Vignettes  3  and  4). 
Participants  in  the  Co  Cdr  (M=22.13,  579=16.70)  and  PL  (M=22.94,  SD=26.15)  roles  were 
equally  active  during  multiple-source  vetting  sessions,  refining  and  deleting  similar  numbers  of 
TTP.  The  variability  between  vignettes  seen  in  Figure  8  is  consistent  with  the  patterns  seen 
between  single-source  sessions  in  Figure  7.  An  examination  of  the  months  of  service  for 
participants  in  multiple-source  vetting  sessions  revealed  that  PLs  (M=  207.20,  579=82.99)  had 
more  military  experience  on  average  than  Co  Cdrs  (M=  1 1 1.50,  579=37.48).  This  led  the  authors 
to  conclude  that  length  of  military  service  plays  an  important  role  in  vetting  productivity,  and 
experience  can  potentially  counteract  the  effects  of  role  assignment. 
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Figure  8.  Average  percent  (and  SD)  of  TTP  modified  or  deleted  during  multiple-source  vetting 
sessions,  by  participant  role. 
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Conclusion.  While  Co  Cdrs  were  sometimes  more  active  during  single-source  vetting 
sessions,  the  effect  was  not  consistent  across  all  sessions.  There  was  some  evidence  to  suggest 
that  length  of  military  service  plays  a  role  in  how  active  a  participant  was  during  a  vetting 
session.  In  multiple-source  sessions,  role  had  less  of  an  effect  on  vetting  activity  than  length  of 
military  service.  Future  researchers  should  consider  military  experience  as  an  important  factor 
contributing  to  vetting  perfonnance. 


Saturation  Effects 


Development  of  TTP 

To  determine  the  point  of  saturation  (Guest,  Bunce,  &  Johnson,  2006)  for  developing 
new  TTP,  we  calculated  the  proportion  of  TTP  items  generated  during  a  development  session 
that  did  not  duplicate  items  generated  in  a  previous  development  session.  For  the  purpose  of  the 
current  project,  the  criterion  for  saturation  was  defined  at  5%.  That  is,  when  the  number  of  new 
TTP  generated  during  a  session  fell  below  5%  the  point  of  saturation  was  reached.  There  was  no 
absolute  justification  for  choosing  5%  as  the  point  of  saturation.  Since  it  is  unlikely,  and 
impractical,  that  all  possible  TTP  will  be  developed  in  an  infinite  number  of  sessions,  future 
research  teams  must  balance  the  relative  costs  and  benefits  associated  with  the  KE  process  to 
determine  their  own  criterion  for  saturation. 

As  shown  in  Figure  9,  by  the  third  development  session  only  about  8%  of  the  TTP 
generated  were  new  TTP.  During  the  fourth  and  fifth  development  sessions  the  percent  of  new 
TTP  dropped  to  2.5%  and  4.5%,  respectively.  Thus,  under  the  conditions  of  this  research  project 
saturation  was  reached  after  three  development  sessions.  Future  research  should  define  the  point 
of  saturation  according  to  the  objectives  of  the  project  and  weigh  the  relative  importance  of  any 
new  TTP  against  the  resources  required  to  generate  them.  For  example,  the  point  of  saturation 
might  be  set  higher  for  TTP  related  to  battlefield  conditions  and  when  lives  are  at  greater  risk, 
and  set  lower  for  TTP  associated  with  less  critical  functions. 

Vetting  of  TTP 

Single-source  vetting  sessions.  To  determine  when  the  point  of  saturation  is  reached  for 
vetting  TTP  during  single-source  vetting  sessions,  the  participants  were  asked  to  rate  the  TTP 
summaries  generated/vetted  for  each  vignette  on  a  100-point  scale,  with  higher  scores  signaling 
better  TTP.  It  is  important  to  note  that  participants  rated  the  TTP  summaries  presented  to  them, 
not  the  resulting  TTP  summaries  they  themselves  generated.  Thus,  for  single-source  vetting 
sessions  the  first  vetting  group  rated  TTP  which  had  yet  to  be  vetted,  the  second  vetting  group 
rated  TTP  which  were  vetted  once,  and  so  on.  We  then  calculated  the  average  ratings  for  TTP 
quality  according  to  the  number  of  times  they  were  vetted — zero,  one,  or  two  times.  As  shown 
in  Figure  10,  after  two  rounds  of  vetting  the  average  ratings  of  the  TTP  reached  95.  In  fact,  after 
two  rounds  of  vetting  81.81%  of  the  ratings  reached  95  or  above.  Thus,  saturation  for  single¬ 
source  vetting  sessions  occurred  after  two  vetting  sessions.  Future  researchers  should  select  a 
saturation  criterion  to  meet  the  goals  of  the  project  and  weigh  the  relative  importance  of  any 
additional  TTP  refinement  against  the  resources  necessary  to  generate  them. 
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Multiple-source  vetting  sessions.  As  with  single-source  vetting,  in  multiple-source 
sessions  the  participants  were  asked  to  rate  the  quality  of  the  received  TTP  summaries  on  a  100- 
point  scale,  with  higher  scores  reflecting  better  TTP.  However,  two  major  differences  between 
multiple-source  and  single-source  vetting  sessions  prevented  us  from  assessing  the  point  of 
saturation  for  multiple-source  vetting  sessions. 

First,  in  single-source  vetting  sessions,  each  participant  received  only  one  TTP  summary 
per  vignette  which  was  previously  vetted  between  zero  and  two  times.  In  the  multiple-source 
vetting  sessions,  each  participant  was  given  five  different  TTP  summaries  to  vet  per  vignette,  one 
from  each  previous  development  session.  The  TTP  summaries  presented  in  multiple-source 
vetting  sessions  had  previously  been  vetted  between  zero  and  three  times.  Thus,  participants  in 
multiple-source  vetting  sessions  received  TTP  summaries  which  were  at  different  levels  of 
maturity  resulting  from  variable  numbers  of  vettings.  The  variance  induced  by  receiving  several 
TTP  at  different  levels  of  maturity  prevented  us  from  calculating  a  measure  of  saturation. 

As  shown  in  Figure  11,  there  were  differences  in  TTP  summary  ratings  across  sessions 
and  across  vignettes,  and  between  participants.  The  large  error  bars  indicate  a  lack  of  agreement 
between  the  participants,  with  some  participants  assigning  a  relatively  high  rating  for  a  TTP 
summary  (above  90),  while  other  participants  assigned  a  low  rating  to  the  same  TTP  summary 
(near  zero).  The  researchers  conducting  the  KE  session  noted  that  the  high  variability  between 
participant  ratings  might  reflect  some  participants  not  “buying  into”  the  procedure  and  remaining 
disengaged  during  the  session. 
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Figure  11.  Average  ratings  (and  SD)  of  TTP  summaries  by  vignette  and  point  of  origin. 
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Second,  each  vignette  was  subjected  to  only  one  multiple-source  vetting  session  as 
compared  with  six  during  single-source  vetting  sessions.  Since  the  TTP  summaries  produced  for 
each  vignette  underwent  only  one  multiple-source  vetting  session,  we  could  not  estimate  the 
point  of  saturation  (i.e.,  ratings  across  sessions). 

Quality  and  Origin  of  Final  TTP 

This  subsection  discusses  characteristics  of  the  final  TTP  that  resulted  from  the  trial 
implementations.  The  information  is  presented  to  assist  future  developers  in  determining  the 
appropriate  mix  of  original  TTP  development  and  single-source  vetting  sessions,  as  well  as 
participant  group  composition,  sufficient  to  maximize  return  on  investment. 

Quality  of  Final  TTP 

It  would  be  naive  to  assume  that  all  participants  would  universally  rate  any  set  of  TTP  as 
perfect.  However,  the  current  methodology  was  able  to  produce  TTP  which  received  perfect  or 
near  perfect  ratings  across  several  vetting  sessions  by  experienced  military  personnel  (see 
Figures  10  and  1 1).  The  TTP  were  then  examined  independently  by  two  SMEs  with  over  20 
years  of  military  experience,  as  they  assembled  a  final  set  of  TTP.  The  SMEs  reduced  the 
number  of  final  TTP  to  construct  a  comprehensive  and  congruent  set  of  TTP  for  each  vignette. 
The  final  TTP  for  each  vignette  included  a  summary  statement  comprised  of  6  to  14  sentences, 
followed  by  separate  statements  for  METT-TC.  The  final  numbers  of  summary  and  METT-TC 
statements  for  each  vignette  were  distributed  as  follows:  Vignette  1,  50  statements;  Vignette  2, 
36;  Vignette  3,  40;  and  Vignette  4,  31  statements. 

Origin  of  Final  TTP 

To  develop  a  benchmark  for  the  number  of  development  and  vetting  sessions  needed,  we 
tracked  the  point  of  origin  and  vetting  refinements  for  each  item  of  the  final  TTP.  Figure  12 
depicts  the  point  of  origin  for  TTP  according  to  session  and  vignette.  Approximately  70%  of  the 
final  TTP  were  developed  during  development  session  one  (M=42.43%,  SD=  8.52%)  and 
development  session  two  (M=26.57%,  SD= 8.83).  Following  the  second  session  there  was  a 
marked  decline  in  the  amount  of  final  TTP  produced.  The  third  and  fourth  development  sessions 
produced  small  contributions  (M=7.07%,  £0=1.05  andM=7.18%,  50=1.86,  respectively)  to  the 
final  TTP,  as  did  session  five  (M=4.32%,  50=3.77).  Participants  in  vetting  sessions  also 
generated  some  original  TTP,  accounting  for  a  moderate  proportion  (47=12. 42%,  50=7.12)  of 
the  final  TTP. 
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Figure  12.  Percent  of  final  TTP  generated  according  to  session  number  and  type  across  vignettes. 
Origin  of  Vetting  Refinemen  ts 

Nearly  one-quarter  (23.57%)  of  the  final  TTP  were  unrefined  from  their  point  of  origin. 
That  is,  the  final  statement  appeared  exactly  as  it  was  proposed  by  the  first  person/group  that 
developed  it.  The  majority  of  the  remaining  final  TTP  were  refined  either  once  (43.3 1%)  or 
twice  (29.30%),  with  only  a  small  percentage  (3.19%)  being  refined  three  times.  Figure  13 
displays  the  percent  of  final  TTP  refined  through  vetting  according  to  vignette  and  vetting 
session.  Nearly  75%  of  the  refinements  occurred  during  either  the  first  single-source  vetting 
session  (M=43.45%,  SD=  6.48)  or  the  multiple-source  vetting  session  (M= 30.59,  579=6.41). 
Vetting  productivity  began  to  drop  dramatically  between  the  second  single-source  session 
(M=19.89,  579=3.68)  and  the  third  single-source  session  (M= 6.07,  579=1.69).  These  results 
indicate  that  single-source  vetting  may  hit  the  point  of  diminishing  returns  after  two  sessions. 
Additional  benefits  may  occur  when  a  multiple-source  session  is  conducted  as  a  follow-up  to  two 
single-source  sessions.  Once  again,  it  is  ultimately  up  to  the  KE  team  members  to  conduct  a 
cost/benefit  analysis  to  determine  if  the  additional  infonnation  gained  by  conducting  a  third 
single-source  session  is  worth  the  time  and  resources. 

The  large  number  of  items  refined  in  the  multiple-source  sessions  may  reflect  both 
previously  unvetted  items  produced  in  development  sessions  4  and  5,  and  any  new  TTP 
developed  during  the  final  single-source  vetting  session.  Alternatively,  the  spike  in  number  of 
TTP  modified  may  be  a  byproduct  of  the  multiple-source  comparison  process.  Providing 
participants  with  a  larger  and  more  diverse  set  of  TTP  may  enable  them  to  identify  the  best 
aspects  of  the  various  TTP  and  produce  refinements  that  reflect  “the  cream  of  the  crop.”  In 
support  of  this  second  alternative,  the  data  reveal  that  78.57%  of  the  refinements  made  to  the 
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final  TTP  during  the  multiple-source  sessions  were  revisions  of  previously  vetted  TTP,  with  only 
21.43%  of  the  refinements  occurring  on  previously  unvetted  TTP.  Thus,  there  is  a  substantial 
benefit  of  allowing  participants  to  conduct  cross  comparisons  during  multiple-source  sessions. 


60 


O) 


One 


□  Single  Source  First  Vetting 

■  Single  Source  Second  Vetting 

□  Single  Source  Third  Vetting 


Two  Three  Four 


Vignette 


Figure  13.  Percent  of  final  TTP  modified  through  vetting  according  to  session  and  vignette. 
Influence  of  Demographics 

Some  of  the  data  suggest  that  participants’  military  background  influenced  the  TTP 
development.  The  third  development  group  showed  a  decline  in  total  TTP  production,  new  TTP 
production,  and  TTP  quality  as  measured  by  the  percent  of  final  TTP  issuing  from  that 
development  session.  While  some  of  the  decline  is  no  doubt  attributable  to  saturation  effects, 
some  is  likely  due  to  the  participants’  military  background.  Three  of  the  four  personnel  who 
participated  in  development  session  3  were  supply  personnel.  Only  the  participant  who  served  in 
the  Co  Cdr  role  had  military  experience  outside  of  supply.  The  sharp  decline  may  in  part  reflect 
the  lack  of  relevant  expertise  by  those  participants.  Having  participants  with  appropriate 
expertise  is  an  enabler  for  more  fruitful  TTP  development  sessions. 

The  results  on  the  quality  and  origins  of  the  final  TTP  shed  light  on  the  dynamics  and 
effectiveness  of  the  KE  process.  Key  points  for  future  developers  include: 

•  The  overall  quality  of  the  final  TTP  was  rated  highly  by  SMEs. 

•  Approximately  70%  of  the  information  that  ultimately  appeared  in  the  final  TTP 
originated  from  the  first  two  TTP  development  sessions. 

•  Single-source  vetting  may  hit  the  point  of  diminishing  returns  after  two  sessions. 

•  Nearly  75%  of  all  TTP  refinements  occurred  during  the  first  single-source  vetting 
session  or  during  the  multiple-source  vetting  session. 
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The  background  and  experience  of  the  participants  affect  the  quality  of  the  KE  process 
outcomes. 


Lessons  Learned 

Lessons  learned  were  derived  from  written  input  provided  by  researchers  and  participants 
on  survey-style  feedback  forms.  Table  8  summarizes  the  major  lessons  for  the  FLEX  method, 
organized  under  five  categories  of  interest. 

Table  8 

Summary  of  Lessons  Learned  by  Category 


Category 

Orientation 


Simulation 

Timing 


Participants 

Procedures 


Lesson  Learned 

Orientation  materials  should  describe  the  KE  goals,  KE  process,  and  simulation. 

A  practice  vignette  substantially  enhances  readiness  of  participants. 

Guidelines  for  recording  technical  problems  should  be  provided  to  operators. 

Simulation  software  should  provide  a  stable,  realistic,  interactive  environment. 

The  simulation  should  realistically  represent  the  features/functioning  of  the  target  system. 
Simulation  software  should  be  sufficiently  mature  to  minimize  technical  problems. 
Sufficient  time  should  be  allocated  to  troop  leading  procedures  prior  to  mission 
execution. 

A  half  hour  is  sufficient  for  executing  simple  company-platoon  missions. 

The  daily  schedule  should  be  sufficiently  flexible  to  accommodate  delays  and  over-runs. 
A  daily  schedule  of  7  to  8  hours  is  sufficient  for  four  vignette  cycles. 

A  daily  schedule  less  than  7  to  8  hours  would  be  insufficient. 

Sufficient  time  is  needed  between  sessions  to  produce  and  check  vetting  materials. 
Motivated,  fully  qualified  participants  greatly  enhance  the  quality  of  the  KE  products. 
Participants’  backgrounds  should  be  matched  to  the  subject  matter  being  investigated. 
Developing  vignette-specific  TTP  is  both  feasible  and  necessary. 

Knowledgeable  and  experienced  SMEs  working  on  the  research  team  are  imperative. 
Participant  role  assignment  affects  the  data. 

Multiple  development  sessions  are  beneficial,  but  with  diminishing  returns. 

Conducting  both  single-source  and  multiple-source  vetting  sessions  is  beneficial. 

Direct  database  entry  by  participants  can  expedite  data  analysis  and  reduce  data  loss. 
Video-taping  sessions  would  generally  be  preferable  to  audio  recording. _ 


Orientation 

Participant  performance  during  the  first  vignette  was  greatly  enhanced  by  providing  a 
brief  practice  session  as  part  of  the  orientation.  The  practice  session  familiarized  the  participants 
with  the  apparatus,  thinking  aloud,  and  the  verbal  directions  required  to  instruct  the  simulation 
operator.  Orientation  materials  should  be  provided  to  participants  far  enough  in  advance  to 
impart  a  frame  of  reference  for  the  goals  of  the  KE  process,  the  procedures,  and  the  specific 
vignette  missions.  To  ensure  adequate  study  time,  the  facilitators  may  choose  to  deliver  the 
orientation  materials  to  participants  at  least  one  day  in  advance. 
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Simulation 


High  quality  simulations  play  an  essential  role  in  producing  “buy-in”  from  participants. 
Stable,  environmentally  rich,  systems-capable  simulations  on  par  with  or  superior  to  home  video 
game  simulations  are  needed  to  effectively  immerse  the  participants  in  the  tactical  exercises. 

Timing 


The  daily  schedule  provided  enough  time  to  complete  the  established  goals  without 
overtaxing  the  participants,  facilitators  or  operators.  However,  sufficient  time  needs  to  be 
scheduled  between  sessions  to  pennit  the  KE  team  to  process  data  and  prepare  materials  for 
vetting  sessions.  Hastily  revised  materials  led  to  incorrectly  fonnatted  forms  and  scales,  missing 
items,  and  the  inclusion  of  incorrect  items. 

Participants 

Participants  lie  at  the  heart  of  any  KE  session.  Participants  with  military  backgrounds 
congruent  with  the  FCS  are  necessary  to  maximize  the  effectiveness  of  the  KE  process.  At  the 
same  time,  having  FCS-congruent  military  backgrounds  is  not  sufficient  to  guarantee  high 
quality  TTP.  A  motivated,  insightful,  and  expressive  participant  is  equally  important. 

Procedures 

A  KE -based,  simulation-driven  FLEX  method  can  effectively  and  efficiently  produce 
TTP  for  FCS  capabilities.  Between  two  and  three  development  sessions,  two  single-source  and 
one  multiple-source  vetting  session  should  be  sufficient.  Production  and  vetting  of  TTP  were 
affected  by  echelon  (role  assignment).  Increasing  the  number  of  echelons  represented  in  future 
TTP  development  sessions  may  enhance  FCS  employment  by  generating  broader  cross-echelon 
TTP.  Some  loss  of  data  occurred  due  to  difficulties  with  reading  handwriting  and  interpreting 
notations.  Having  participants  enter  their  responses  directly  into  computers  would  assist  with 
TTP  turnaround,  source  tracking,  avoidance  of  data  loss  and  errors,  and  electronic  backup. 

Methodology  Improvements 

Our  team  made  one  substantive  modification  of  the  original  FLEX  method  by  formally 
incorporating  multiple  starting  points  through  separate  initial  TTP  development  sessions.  While 
the  original  method  employed  a  single  point  of  origin  for  development,  we  utilized  a  multiple- 
source  approach.  These  multiple  tracks  of  development  then  converge  into  a  consensus  building 
session  that  includes  a  multiple-source  vetting  session  and  final  TTP  refinement,  ft  is  important 
to  note  that  this  modification  was  not  outside  the  original  theoretical  framework  of  the  FLEX 
methodology,  but  a  modification  in  how  the  methodology  was  implemented.  We  adopted  the 
multiple-source  process  over  concerns  that  the  outcome  of  a  single-track  approach  could  be 
limited  by  the  simple  fact  of  its  narrow  origins.  Theoretically,  multiple  starting  points  would  (a) 
lead  to  a  variety  of  distinct  solutions,  each  valuable  and  mature  in  its  own  right,  or  (b)  confirm 
the  validity  of  a  single,  common  outcome  achieved  with  multiple,  independent  sources.  Figure 
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14  illustrates  a  multiple-track  framework  with  each  track  originating  at  an  independent  starting 
point. 
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Figure  14.  A  multiple-track  FLEX  process. 

For  the  purposes  of  this  research,  the  initial  development  tracks  numbered  five.  For 
future  implementation,  we  recommend  a  minimum  of  two  development  tracks,  each  followed  by 
a  minimum  of  one  single-source  vetting  session,  yielding  a  “2-by-2”  matrix  as  shown  in  Figure 
14  above.  Depending  upon  the  TTP  developer’s  expectations  and  objectives,  as  well  as  the 
availability  of  resources  to  support  TTP  development,  the  matrix  can  be  expanded  vertically  or 
horizontally  to  achieve  greater  resolution,  potential  variety,  and/or  more  confidence  in  the 
results.  Figure  15  illustrates  how  expanding  the  matrix  could  impact  outcomes. 
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Figure  15.  Conceptual  outcomes  of  an  expanded  FLEX  process. 

Our  implementation  approach  intentionally  kept  each  TTP  track  separate  until  the  final 
consensus-building  step,  in  theory  to  allow  each  set  of  TTP  to  fully  mature  without  external 
influence.  Alternatively,  the  outputs  from  each  KE  session  could  be  shared  across  tracks, 
effectively  making  every  vetting  session  a  multiple-source  session.  While  this  variation  has 
merit,  we  did  not  test  it  during  this  effort. 

In  all  cases,  the  final  step  of  consolidating  and  integrating  previous  outputs  is  essential  to 
produce  one  coherent,  complete,  and  feasible  TTP  or  solution. 

CONCLUSIONS  AND  RECOMMENDATIONS 

Future  forces  empowered  with  FCS  will  require  streamlined,  cost  effective,  and  time 
efficient  methods  for  developing  new  TTP.  The  primary  goal  of  the  current  research  was  to 
develop  a  simulation-based  KE  method  for  forging  TTP  and  to  assess  the  effectiveness  of  the 
methodology  as  a  general  framework  for  future  TTP  development.  The  results  indicate  that  the 
KE-based,  simulation-driven  FLEX  method  can  produce  effective  TTP  as  rated  by  Soldiers.  The 
FLEX  method  enables  researchers  to  gain  the  perspective  of  how  the  real  users,  the  Soldiers  in 
the  field,  would  employ  the  technologies  in  real-world  missions.  By  working  with  Soldiers  from 
different  MOSs,  as  well  as  levels  and  types  of  field  experience,  investigators  can  develop  a  rich 
perspective  of  not  only  how  these  new  technologies  would  be  perceived  but  also  how  they  would 
be  used,  adapted,  and  advanced  by  Soldiers. 

Future  forces  will  require  integrated  TTP  to  guide  the  employment  of  FCS  capabilities. 
Coordinated,  cohesive  actions  across  and  within  echelons  will  be  imperative.  In  this  project, 
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Soldiers  with  different  MOSs  typically  employed  the  UAS  in  a  manner  reflecting  their  own 
MOS,  assigned  role,  level  of  expertise,  and  type  of  experience.  Elicitation  processes  that  include 
multi-echelon  role  players  will  facilitate  the  coordinated  efforts  that  future  forces  need. 
Perspectives  from  several  echelons  and  unique  applications  should  be  represented  in  the  final 
TTP,  providing  both  specific  and  generalized  guidelines  for  employing  FCS  capabilities. 

Established  METT-TC  factors  will  remain  relevant  after  the  advent  of  FCS-enabled 
operations.  The  manner  in  which  future  forces  handle  METT-TC  factors  will  be  greatly  affected 
by  FCS  technologies  that  dramatically  enhance  warfighting  capabilities.  With  the  expected 
number  of  improvements,  innovative  approaches  will  be  needed  to  develop  and  test  the  new 
technologies.  Simulation  vignettes  provide  a  stable,  cost  effective  environment  to  explore 
concepts  that  involve  complex  problem  solving.  They  are  capable  of  presenting  many  of  the 
same  types  of  challenges  as  real-world  environments  in  real  time.  In  fact,  simulations,  especially 
simulation-based  games,  are  powerful  learning  tools  because  they  enable  learners  to  interact  with 
and  engage  with  environments  that  pose  complex,  ill-structured  problems.  Learners  are  able  to 
visualize,  interact  with,  and  manipulate  systems  and  variables  within  a  complex,  authentic 
environment  (Gredler,  2004). 

The  flexible  methodology  presented  in  this  report  enables  effective  TTP  development 
sessions  to  support  future  forces.  The  report  outlines  a  CTA-based  KE  protocol  that  has  been 
shown  capable  of  producing  effective  TTP  for  FCS.  It  provides  benchmarks  for  the  number  and 
type  of  KE  sessions  required  to  develop  effective  TTP.  Naturally,  future  researchers  and 
developers  will  always  need  to  consider  their  specific  requirements  when  creating  a  TTP 
development  plan.  They  must  take  into  account  their  goals,  the  complexity  of  the  system  being 
assessed,  the  blending  of  current  and  FCS,  the  likely  level  of  participants’  expertise,  and  other 
considerations. 

This  report  provides  a  “proof  of  concept”  and  general  framework  for  conducting  KE 
sessions  to  develop  TTP  for  future  capabilities.  The  following  recommendations  are  offered  to 
promote  effective  utilization  and  expansion  of  the  findings. 

•  ARI  should  take  steps  to  disseminate  the  findings  of  this  report  and  make  the 
information  readily  available  to  Future  Force  developers. 

•  The  Developer's  Support  Package  should  be  expanded  to  include  a  comprehensive 
user's  guide.  The  package  should  be  made  available  to  researchers  and  TTP 
developers. 

•  Researchers  must  carefully  select  a  suitable  simulation  to  support  TTP  development 
using  the  FLEX  method.  Mature  traditional  simulations  or  even  modified  game- 
based  commercial  applications  may  offer  advantages  over  immature  simulation 
software. 

•  Future  TTP  developers  must  screen  for  and  select  experienced,  insightful,  expressive 
participants  to  optimize  TTP  production. 
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•  Future  developers  should  ensure  that  participants  understand  what  comprises 
effective  TTP,  including  the  level  of  detail  required. 

•  Additional  research  should  be  conducted  to  examine  TTP  development  using  the 
FLEX  method  at  higher  echelons  (e.g.,  battalion  level). 

•  Researchers  should  explore  alternative  ways  to  examine  and  evaluate  the  quality  of 
TTP  developed  during  simulation-based  KE  sessions. 

•  Future  research  should  be  conducted  to  determine  whether  TTP  could  be  developed 
simultaneously  for  multiple  FCS  systems/capabilities  during  the  same  KE  session  to 
accelerate  the  development  of  cohesive  TTP  for  the  Future  Force. 
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Appendix  A 


AETF 

ARI 

C2 
Cdr 
CGF 
Co  Cdr 
CTA 

FCS 

FFID 

FLEX 

FRAGO 

KE 

M 

MAPEX 

METT-TC 

MOS 

OF-OOS 

OOS 

OPORD 

PC 

PL 

SD 

SME 

S03 

SOP 

TEWT 

HP 


Acronyms 

Army  Evaluation  Task  Force 

U.S.  Anny  Research  Institute  for  the  Behavioral  and  Social  Sciences 

Command  and  Control 
Commander 

Computer  Generated  Force 
Company  Commander 
Cognitive  Task  Analysis 

Future  Combat  Systems 
Future  Force  Integration  Directorate 
Flexible  Method  of  Cognitive  Task  Analysis 
Fragmentary  Order 

Knowledge  Elicitation 

Mean 

Map  Exercise 

Mission,  Enemy,  Terrain/Weather,  Troops,  Time,  and  Civilians 
Military  Occupational  Specialty 

Objective  Force-OneSAF  Objective  System 
OneSAF  Objective  System 
Operation  Order 

Personal  Computer 
Platoon  Leader 

Standard  Deviation 
Subject  Matter  Expert 
Spin  Out  3 

Standing  Operating  Procedure 

Tactical  Exercises  without  Troops 
Tactics,  Techniques,  and  Procedures 


A-l 


TTP-C 

UAS 


Tactics,  Techniques,  Procedures,  and  Considerations 
Unmanned  Aircraft  System 
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Appendix  B 

TTP  Development  Data  Collection  Package 


NOTE  TO  FACILITATOR:  This  package  is  for  use  with  participants  who  are 
conducting  TTP  development  ONLY.  If  you  are  working  with  a  group  that  is 
reviewing/refining  previous  TTPs,  please  use  the  alternate  package  entitled  TTP 
Review/Refinement  Data  Collection  Package. 


About  This  Document 

•  This  package  gives  facilitators  an  integrated  set  of  materials  to  use  in  controlling 
activities  and  capturing  data. 

•  Please  consult  the  basic  Implementation  Plan  for  specific  directions  regarding  the 
overall  KE  process. 

•  The  goal  of  the  KE  process  is  to  focus  participants'  attention  on  relevant  aspects  of  a 
vignette  to  foster  development  of  TTPs.  (A  vignette  is  a  brief  scenari o  designed  to 
run  in  simulation  and  elicit  participants’  thoughts,  actions,  and  decisions). 

•  The  KE  process  involves  multiple  steps  that  take  the  participants  through  a  tactical 
situation  and  capture  their  ideas  regarding  appropriate  TTPs. 

•  The  questions  in  this  protocol  are  designed  to  elicit  information  about  cues,  planning 
factors,  critical  decisions,  and  warfighting  TTPs. 

•  This  package  structures  your  note-taking  activities.  Please  record  all  observations, 
comments,  and  key  points  on  the  pages  of  this  package. 

•  Be  sure  to  use  the  Procedural  Problems  Log  to  record  circumstances  that  may  impact 
interpretation  of  the  data,  such  as  missing  participants. 


Date _  Facilitator _ 

Installation _  Group  # _ 

Group  Function:  Dvlpt  /  Vetting  Sub-group  Role: 


Commander  (Cdr)  / 
PLs 


Place  Participants’ 
ID  label(s)  here 
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Step  1 :  Complete  Checklist  and  Equipment  Testing 

NOTE  TO  FACILITATOR:  Be  sure  to  complete  this  step  early  enough  to  resolve  any  issues 
prior  to  participant  arrival. 

FLEX  TTP  Checklist  for  Materials  and  Equipment  Testing 

General  Materials 

_ Three-hole  punch 

_ Watch  or  clock 

_ Backup  drive  for  audio  files 

_ Software  for  audio  recorders 

_ Digital  camera  capable  of  downloading  images  via  Universal  Serial  Bus  cable 

_ Digital  audio  recorders  (2-4),  batteries  and  microphones;  recorder-to-PC  cable 

_ Previous  files  downloaded 

_ Tested,  volume  adjusted 

Materials  for  Technical  Support  Team 

_ Technical  Problems  Log  (hardcopy) 

_ Five  networked  OneSAF  objective  system  (OOS)  workstations 

_ Tactical  Vignette  Materials  including  OOS  files 

_ Vignettes  cued,  volume  adjusted 

Materials  for  Participants 

_ Tactical  communication  emulators  tested  and  volume  adjusted 

_ Poster  paper/easel  for  participants  to  use  in  developing  TTPs 

_ Pens  and/or  markers 

_ Participant  Binders  (x4),  each  containing: 

_ Tactical  Vignette  Materials  (hardcopy) 

_ Orientation  (project  overview,  FCS/Spin  Out  (SO)3  intro,  UAS 

capabilities,  etc.) 

_ Schedule  of  events 

_ Participant  Profile  Questionnaire 

_ Sample  TTP 

_ TTPs  from  previous  sessions  (Refinement  groups  only) 

_ Participant  worksheets  for  writing  TTPs 

_ Participant  Feedback  Questionnaire 

_ Blank  paper  for  writing  notes 

Materials  for  Facilitators 

_ Master  Participant  Chart  (one  hardcopy) 

_ Participant  ID  labels  (4  sets) 

_ Placed  on  appropriate  booklets  and  forms 

_ Session  Schedule  (x2) 

_ Knowledge  Elicitation  Protocol  (x2) 

_ Data  Collection  Package  (x2) 

_ Procedural  Problems  Log  (x2) 
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Step  2:  Welcome  Participants 


1 .  WELCOME:  Exercise  director  says,  “  Welcome  to  the  Army  Research  Institute 's  future- 

focused  TTP  development  project.  My  name  is _ and  my  associates  are _ , _ , 

and _ .  The  goal  of  this  project  is  to  develop  tactics,  techn  iques,  and  procedures,  or 

TTPs,for  employing  an  FCS  unmanned  aircraft  system,  or  UAS.  You  will  learn  more  about 
UASs  in  your  orientation  materials.  Part  of  ARI’s  research  program  supports 
transformation.  To  develop  the  UAS  TTPs,  we  are  asking  you  to  role-play  your  assigned 
position  in  several  scenario-based  simulation  vignettes.  While  watching  a  vignette  play  out 
on  a  computer  workstation,  you  will  direct  the  workstation  operator  on  the  course  of  action 
to  take  for  a  given  situation.  Simultaneously,  we  want  you  to  share  with  us  aloud  your 
thoughts  and  decisions  regarding  a  chosen  course  of  action  and  how  a  UAS  could  best  be 
employed  to  assist  you  in  completing  your  mission.  We  may  ask  you  questions  from  time  to 
time.  Overall,  the  session  will  take  approximately  8  hours  of  your  time.  We  would  like  to 
thank  you  upfront  for  your  participation  in  this  important  project.  Our  ultimate  goal  is  to 
enhance  force  effectiveness  in  the  Global  War  on  Terror.  Do  you  have  any  questions?  ” 

If  you  open  your  binder  you  will  see  a  brief  overview  of  today ’s  schedule: 


Schedule 

Tasks 

0830-0900 

Welcome 

0900-1030 

Orientation/Practice  Session 

1030-1130 

Complete  Vignette  1  (break) 

1130-1230 

Complete  Vignette  2  (break) 

1230-1330 

LUNCH 

1330-1430 

Complete  Vignette  3  (break) 

1430-1530 

Complete  Vignette  4  (break) 

1530-1630 

Summary  Review 

1630-1645 

Wrap  Up 

“We  will  have  regular  breaks,  and  the  location  of  the  restrooms  are _ .  If  you  have  other 

specific  needs,  please  let  us  know.  ” 
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Step  3:  Complete  Participant  Profile  Questionnaire 


INSTRUCTIONS  TO  FACILITATOR:  Familiarize  the  participants  with  their 
binders.  Start  with  the  list  of  contents  and  talk  them  through  the  Tabs,  including 
the  Worksheets  and  blank  paper  at  the  end.  Answer  any  questions. 

Instructions  to  Participants:  “ Please  turn  to  the  Participant  Profile  Questionnaire  in  your 
binder.  This  questionnaire  asks  you  for  information  on  specific  types  of  training  and  experiences 
you  have  had  in  the  Army.  Please  respond  with  as  much  detail  as  you  can,  and  let  me  know  if 
you  have  questions.  ’’ 

NOTE  TO  FACILITATOR:  Be  sure  to  review  participant  responses  to  verify  that 
the  information  is  complete.  If  some  of  their  responses  are  incomplete,  ask  them 
to  respond  orally  and  record  their  responses. 

Step  4:  Orientation 

NOTE  TO  FACILITATOR:  The  Vignette  and  Orientation  materials  in  the 
participant  binders  are  for  their  use  at  any  time.  The  materials  include  tactical 
materials  (e.g.,  OPORD,  maps,  FRA&Os),  project  overview,  FCS/S03  introduction, 
list  of  UAS  capabilities,  agenda,  etc. 

Instructions  to  Participants:  “The  materials  in  your  binder  are  available  for  your  use  as  you 
work  through  the  simulation  vignettes.  Right  now,  we  want  you  to  view  the  multimedia 
Orientation  on  your  workstation  and  ask  questions  as  you  go  along.  ” 

The  workstation  operators  help  each  participant  start  the  Orientation  and  answer  questions  as 
they  arise.  After  the  last  participant  finishes,  the  exercise  director  asks  for  questions. 

Once  all  participants’  questions  have  been  answered,  the  exercise  director  provides  the  following 
directions  for  the  vignettes. 

a.  “ You  are  going  to  watch  several  simulation  vignettes  in  which  an  FCS  UAS  could  be 
employed.  As  you  role-play  your  assigned  position,  please  describe  your  reactions, 
thoughts,  and  decisions  as  you  work  to  complete  the  mission  while  you  direct  the 
workstation  operator.  We  are  supplying  devices  to  emulate  tactical  communication.  We 
will  be  audio  recording  your  verbal  responses  and  analyzing  the  content.  It  is  important 
that  you  think  aloud  as  you  work  and  try  to  explain  your  decisions  and  actions 
throughout  the  vignette. 

b.  Feel  free  to  consult  the  materials  in  your  binders  as  needed.  There  is  no  time  limit,  but 
please  move  along  in  order  to  stay  on  schedule. 

c.  Immediately  following  each  vignette,  we  will  conduct  a  brief  review  and  ask  you  to 
develop  (or  review  and  vet)  some  TTPs. 
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i.  For  those  participants  who  are  developing  TTPs:  “You  have  a  sample  set  of  TTPs  in 
your  hinder  materials.  Please  use  them  as  a  model  to  develop  your  own.  Take  a 
moment  to  review  them  now  before  we  begin.  ”  (See  Step  1  of  KE  Protocol) 

ii.  For  those  participants  who  are  reviewing/refining  TTPs:  “You  have  a  set  of 
previously  developed  TTPs  for  UASs  in  your  binder  materials.  You  will  be  reviewing 
and  refining  these  based  on  your  own  experience  in  the  simulation  vignettes.  Take  a 
moment  to  look  over  these  now  before  we  begin.  ”  (See  Step  1  of  KE  Protocol) 

d.  Do  you  have  any  questions?  (Facilitators  respond  to  questions.) 

Step  5:  Practice  Vignette 

Instructions  to  Participants:  To  familiarize  you  with  the  vignettes,  we  ’re  going  to  conduct  a 
practice  event.  Feel  free  to  ask  questions  as  we  go  along.  As  you  execute  your  assigned  role, 
remember  to  verbalize  your  thoughts  and  actions  so  we  know  what  you  ’re  doing  and  why.  When 
I  ask  a  question,  try  to  answer  it  as  best  you  can.  Please  be  thinking  about  the  TTPs  that  would 
be  appropriate,  even  though  we  won ’t  be  developing  TTPs  for  this  practice  vignette. 

NOTE  TO  FACILITATOR:  Let  the  participant(s)  get  a  feel  for  the  simulation  for 
the  f irst  10  minutes,  then  start  asking  questions  from  the  list  below.  Help  them 
understand  the  simulation  environment  and  their  role  in  the  TTP  development 
process.  Record  notes  about  things  you  want  to  explain  at  the  end  of  the  practice 
event. 

1 .  What  are  you  paying  attention  to  in  this  situation?  (observations/cues) 

2.  What  factors  are  you  considering  as  you  plan  a  course  of  action?  (factors) 

3.  What  is  the  best  course  of  action  to  take  at  this  point  and  why?  (key  decision) 

4.  What  is  influencing  the  approach  you  take?  (tactic) 

5.  What  technique/s  are  you  using  at  this  point  to  achieve  the  mission?  (technique) 

6.  Is  there  a  specific  procedure  that  you  would  follow  at  this  point?  (procedure) 


At  the  end  of  the  practice  event : 

•  Facilitator  asks  the  participant(s)  what  questions  they  have  about  the  simulation,  the  vignette, 
their  tactical  role,  or  thinking  aloud. 

•  Facilitator  uses  notes  made  during  the  vignette  to  clarify  or  explain  how  the  participant(s) 
should  be  role-playing  and  thinking  aloud. 

•  Give  the  participant(s)  a  5 -minute  break  while  the  research  team  prepares  for  the  first  TTP 
development  run. 
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Step  6:  Think-Aloud  Prompts  for  Vignette  1 

Facilitator's  Action:  Turn  on  audio  recorder!  State  date,  installation,  group 
number,  and  vignette  number. 

Instructions  to  Participants:  We  are  now  ready  to  work  through  the  first  simulation  vignette. 

Your  job  is  to  role-play  in  character  and  verbalize  your  thoughts  and  actions  so  we  can  better 
understand  your  warfighting  process.  I  may  ask  you  a  few  questions  to  help  you  think  aloud.  If 
you  find  it  hard  to  respond  to  a  question  while  you  are  engaged  with  the  simulation,  continue 
working  through  the  vignette  and  answer  at  a  later  time  that  works  best  for  you. 

NOTE  TO  FACILITATOR:  Be  sure  to  cycle  through  each  of  these  questions  as  the 
participant  reaches  key  decision  points.  Record  the  participant's  responses  to  each 
question  on  this  form.  While  interactions/dialogue  will  be  audio-recorded,  having 
your  notes  will  speed  the  data  analysis  process.  Make  sure  recorder(s)  are  on! 

1.  What  are  you  paying  attention  to  in  this  situation?  (observations/cues) 

2.  What  factors  are  you  considering  as  you  plan  a  course  of  action?  (factors) 

3.  What  is  the  best  course  of  action  to  take  at  this  point  and  why?  (key  decision) 

4.  What  is  influencing  the  approach  you  take?  (tactic) 

5.  What  technique/s  are  you  using  at  this  point  to  achieve  the  mission?  (technique) 

6.  Is  there  a  specific  procedure  that  you  would  follow  at  this  point?  (procedure) 
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KE  Think-Aloud  Data  Collection  Form 


Vignette  1 

Think  Aloud  Data 

1.  Observations/Cues 

2.  Factors  Considered 

3.  Key  Decision 

4.  Tactic 

5.  Technique 

6.  Procedure 

1.  Observations/Cues 

2.  Factors  Considered 

3.  Key  Decision 

4.  Tactic 

5.  Technique 

6.  Procedure 

1.  Observations/Cues 

2.  Factors  Considered 

3.  Key  Decision 

4.  Tactic 

5.  Technique 

6.  Procedure 

Step  7:  Developing  UAS  TTPs  for  Vignette  1 


Instructions  to  Participants:  “In  your  binders  you  will  find  the  sample  TTP  and  a  worksheet  for 
Vignette  1  (Handout).  Please  take  about  5  minutes  and  brainstorm  a  set  of  TTPs  for  employing 
the  UAS.  In  developing  your  TTPs,  you  should  use  the  sample  but  not  be  limited  by  its  format  or 
structure.  Please  develop  a  set  of  TTPs  that  would  enable  you  to  optimally  employ  a  UAS  in  the 
situation  simulated  in  this  vignette.  Also,  please  consider  the  factors  listed  in  your  worksheet  as 
they  relate  to  employing  a  UAS  in  the  situation  simulated  in  this  visnette.  After  you  write  down 
your  ideas,  we  will  discuss  them  together.  ” 

INSTRUCTIONS  TO  FACILITATOR:  When  the  participant(s)  finish  writing, 
discuss  the  details  of  their  TTPs  using  the  following  questions. 

la.  What  are  the  TTPs  you  just  brainstormed  for  this  vignette? 

lb.  How  do  these  TTPs  differ  from  what  you  planned  at  the  start  of  the  vignette? 

2.  How  would  the  following  factors  influence  the  TTPs? 

a.  Time  available 

b.  Troops  available 

c.  Different  types  of  terrain 

d.  Presence  and  location  of  enemy 

e.  Presence  and  location  of  civilians 

f.  Other  factors  identified  by  the  participant s) 

3.  Given  the  mission  you  just  completed,  what  are  some  negative  outcomes  that  might  result 
from  using  a  UAS?  How  could  these  be  avoided? 

4.  Looking  at  your  TTPs  in  light  of  the  factors  just  discussed,  what  would  you  add  or  change? 

5.  What  capabilities  and/or  attributes  does  the  UAS  need  to  support  mission  accomplishment? 

INSTRUCTIONS  TO  FACILITATOR:  Summarize  the  TTPs  for  the  participant(s) 
and  ask  for  final  adjustments.  Then  assemble  all  participants  for  a  short  AAR. 

Facilitator's  Action:  Turn  off  audio  recorder. 
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NOTE:  Materials  for  Steps  8-13  (Vignettes  2-4)  are  identical 
to  those  for  Steps  6-7  (Vignette  1)  and  are  omitted  from  this 
document  for  the  sake  of  brevity. 


You  have  completed  all  vignettes. 


Step  14:  Summary  Review 

INSTRUCTIONS  TO  FACILITATOR:  This  stage  of  the  process  follows 
completion  of  the  last  vignette.  Please  assemble  all  participants  into  one  group  and 
discuss  the  following  questions. 

1 .  How  will  the  use  of  a  UAS  vary  by: 


a.  Different  types  of  missions? 

b.  Time  available? 

c.  Different  types  of  terrain? 

d.  Troops  available? 

e.  Presence  and  location  of  enemy? 

f.  Presence  and  location  of  civilians? 

2.  What  capabilities  should  the  UAS  be  equipped  with  to  allow  it  to  optimally  perform  in  the 
field? 

3.  Should  there  be  some  overlap  from  one  UAS  sweep  to  another?  If  so,  how  much? 

4.  Is  there  a  specific  order  in  which  areas  should  be  observed  by  the  UAS?  If  so,  what  should 
this  order  be? 

5.  What  factors  determine  or  influence  the  surveillance  sequence? 

6.  What  type  of  target  location  method(s)  would  be  optimally  suited  for  a  UAS?  (Examples 
include  grid  coordinates,  polar  coordinates,  shift  from  known  point,  or  other.) 
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7.  Would  there  be  a  difference  in  how  a  UAS  would  be  used  by  a  platoon  versus  a  company? 

8.  What  are  the  key  concepts  a  leader  must  be  aware  of  for  optimal  use  of  the  UAS? 

9.  If  you  were  provided  with  a  UAS,  in  what  situations  would  you  use  it?  In  what  situations 
would  you  hesitate  to  use  it  or  even  avoid  using  it? 

10.  What  else  would  you  consider  in  using  a  UAS?  Why? 

1 1 .  Please  briefly  review  the  TTPs  you  developed  for  each  vignette.  Would  you  make  any 
changes  to  the  TTPs?  If  yes,  what  would  you  change? 

12.  Do  you  have  any  comments  on  the  TTP  development  method  or  this  session  in  general? 


Step  15:  Participant  Feedback  Questionnaire 

Instructions  to  Participants:  “If  you  would  now  turn  in  your  binder  to  the  FLEX  TTP  Participant 
Feedback  questionnaire,  please  fill  out  the  questionnaire.  It  should  take  only  a  few  minutes  but 
we  would  really  appreciate  it  if  you  would  take  your  time  and  provide  us  with  thoughtful 
responses.  ” 

Step  16.  Wrap  Up 

•  Facilitators  collect  all  materials  from  participants. 

•  Exercise  director  answers  final  questions  and  explains  research  payoff. 

•  Exercise  director  thanks  participants  and  releases  them. 

•  KE  facilitators  download  digital  audio  files  to  computer  and  turn  off  recorders. 

•  Facilitators  label  and  file  all  paperwork  in  appropriate  folders. 


End  of  Session 
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Development  Worksheet  (Participant) 


Worksheet:  Developing  UAS  TTPs  for  Vignette  1  (Cordon  &  Search) 

Instructions :  Now  that  you  have  completed  the  vignette,  we  would  like  you  to  write  down  your 
TTPs  for  employing  the  UAS.  The  following  steps  will  help  you  do  this. 


1.  First  examine  the  sample  TTP.  In  developing  your  TTP,  you  should  use  this  model  but  not  be 
limited  by  its  format  or  structure.  Please  write  down  the  TTPs  that  would  enable  you  to 
optimally  employ  a  UAS  in  the  situation  simulated  in  this  vignette. 


The  TTPs  you  develop  should  address  the  following  factors  as  well  as  any  other  factors  you 
consider  to  be  important. 

a.  Time  available 

b.  Troops  available 

c.  Different  types  of  terrain 

d.  Presence  and  location  of  enemy 

e.  Presence  and  location  of  civilians 


2.  Given  the  goal  of  the  mission  you  just  completed,  what  are  some  negative  outcomes  that 
might  result  from  using  a  UAS?  How  might  these  be  avoided? 


3.  What  would  you  add  or  change  at  this  point? 


4.  What  capabilities  and/or  attributes  does  the  UAS  need  to  support  mission  accomplishment? 


NOTE:  Participant  worksheets  for  Vignettes  2-4  are  identical 
to  those  for  Vignette  1  and  are  omitted  from  this  document 
for  the  sake  of  brevity. 
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Appendix  C 

TTP  Revicw/Rcfincment  Data  Collection  Package 


NOTE  TO  FACILITATOR:  This  package  is  for  use  with  participants  who  are 
conducting  Single-Source  or  Multi-Source  TTP  review/refinement  ONLY.  If  you 
are  working  with  a  group  that  is  doing  TTP  Development,  please  use  the  alternate 
package  entitled  TTP  Development  Data  Collection  Package. 


About  This  Document 

•  This  package  gives  facilitators  an  integrated  set  of  materials  to  use  in  controlling 
activities  and  capturing  data. 

•  Please  consult  the  basic  Implementation  Plan  for  specific  directions  regarding  the 
overall  KE  process. 

•  The  goal  of  the  KE  process  is  to  focus  participants'  attention  on  relevant  aspects  of  a 
vignette  to  foster  development  of  TTPs.  (A  vignette  is  a  brief  scenario  designed  to 
run  in  simulation  and  elicit  participants’  thoughts,  actions,  and  decisions). 

•  The  KE  process  involves  multiple  steps  that  take  the  participants  through  a  tactical 
situation  and  capture  their  ideas  regarding  appropriate  TTPs. 

•  The  questions  in  this  protocol  are  designed  to  elicit  information  about  cues,  planning 
factors,  critical  decisions,  and  warfighting  TTPs. 

•  This  package  structures  your  note-taking  activities.  Please  record  all  observations, 
comments,  and  key  points  on  the  pages  of  this  package. 

•  Be  sure  to  use  the  Procedural  Problems  Log  to  record  circumstances  that  may  impact 
interpretation  of  the  data,  such  as  missing  participants. 


Date 


Facilitator 


Installation _  Group  # _ 

Group  Function:  Dvlpt  /  Vetting  Sub-group  Role:  Cdr  /  PLs 


Place  Participants’ 
ID  label(s)  here 
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Step  1 :  Complete  Checklist  and  Equipment  Testing 

NOTE  TO  FACILITATOR:  Be  sure  to  complete  this  step  early  enough  to  resolve  any  issues 
prior  to  participant  arrival. 

FLEX  TTP  Checklist  for  Materials  and  Equipment  Testing 

General  Materials 

_ Three-hole  punch 

_ Watch  or  clock 

_ Backup  drive  for  audio  fdes 

_ Software  for  audio  recorders 

_ Digital  camera  capable  of  downloading  images  via  USB  cable 

_ Digital  audio  recorders  (2-4),  batteries  and  microphones;  recorder-to-PC  cable 

_ Previous  fdes  downloaded 

_ Tested,  volume  adjusted 

Materials  for  Technical  Support  Team 

_ Technical  Problems  Log  (hardcopy) 

_ Five  networked  OOS  workstations 

_ Tactical  Vignette  Materials  including  OOS  files 

_ Vignettes  cued,  volume  adjusted 

Materials  for  Participants 

_ Tactical  communication  emulators  tested  and  volume  adjusted 

_ Poster  paper/easel  for  participants  to  use  in  developing  TTPs 

_ Pens  and/or  markers 

_ Participant  Binders  (x4),  each  containing: 

_ Tactical  Vignette  Materials  (hardcopy) 

_ Orientation  (project  overview,  FCS/S03  intro,  UAS  capabilities,  etc.) 

_ Schedule  of  events 

_ Participant  Profile  Questionnaire 

_ Sample  TTP 

_ TTPs  from  previous  sessions  (Refinement  groups  only) 

_ Participant  worksheets  for  writing  TTPs 

_ Participant  Feedback  Questionnaire 

_ Blank  paper  for  writing  notes 

Materials  for  Facilitators 

_ Master  Participant  Chart  (one  hardcopy) 

_ Participant  ID  labels  (4  sets) 

_ Placed  on  appropriate  booklets  and  forms 

_ Session  Schedule  (x2) 

_ Knowledge  Elicitation  Protocol  (x2) 

_ Data  Collection  Package  (x2) 

_ Procedural  Problems  Log  (x2) 
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Step  2:  Welcome  Participants 


2.  WELCOME:  Exercise  director  says,  “  Welcome  to  the  Army  Research  Institute 's  future- 

focused  TTP  development  project.  My  name  is _ and  my  associates  are _ , _ , 

and _ .  The  goal  of  this  project  is  to  develop  tactics,  techn  iques,  and  procedures,  or 

TTPs,for  employing  an  FCS  unmanned  aircraft  system,  or  UAS.  You  will  learn  more  about 
UASs  in  your  orientation  materials.  Part  of  ARI’s  research  program  supports 
transformation.  To  develop  the  UAS  TTPs,  we  are  asking  you  to  role-play  your  assigned 
position  in  several  scenario-based  simulation  vignettes.  While  watching  a  vignette  play  out 
on  a  computer  workstation,  you  will  direct  the  workstation  operator  on  the  course  of  action 
to  take  for  a  given  situation.  Simultaneously,  we  want  you  to  share  with  us  aloud  your 
thoughts  and  decisions  regarding  a  chosen  course  of  action  and  how  a  UAS  could  best  be 
employed  to  assist  you  in  completing  your  mission.  We  may  ask  you  questions  from  time  to 
time.  Overall,  the  session  will  take  approximately  8  hours  of  your  time.  We  would  like  to 
thank  you  upfront  for  your  participation  in  this  important  project.  Our  ultimate  goal  is  to 
enhance  force  effectiveness  in  the  Global  War  on  Terror.  Do  you  have  any  questions?” 

If  you  open  your  binder  you  will  see  a  brief  overview  of  today ’s  schedule: 


Schedule 

Tasks 

0830-0900 

Welcome 

0900-1030 

Orientation/Practice  Session 

1030-1130 

Complete  Vignette  1  (break) 

1130-1230 

Complete  Vignette  2  (break) 

1230-1330 

LUNCH 

1330-1430 

Complete  Vignette  3  (break) 

1430-1530 

Complete  Vignette  4  (break) 

1530-1630 

Summary  Review 

1630-1645 

Wrap  Up 

“We  will  have  regular  breaks,  and  the  location  of  the  restrooms  are _ .  If  you  have 

other  specific  needs,  please  let  us  know.  ” 
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Step  3:  Complete  Participant  Profile  Questionnaire 

INSTRUCTIONS  TO  FACILITATOR:  Familiarize  the  participants  with  their 
binders.  Start  with  the  list  of  contents  and  talk  them  through  the  Tabs,  including 
the  Worksheets  and  blank  paper  at  the  end.  Answer  any  questions. 

Instructions  to  Participants:  "Please  turn  to  the  Participant  Profile  Questionnaire  in  your 
binder.  This  questionnaire  asks  you  for  information  on  specific  types  of  training  and  experiences 
you  have  had  in  the  Army.  Please  respond  with  as  much  detail  as  you  can,  and  let  me  know  if 
you  have  questions.  ” 

NOTE  TO  FACILITATOR:  Be  sure  to  review  participants'  responses  to  verify  that 
the  information  is  complete.  If  some  of  their  responses  are  incomplete,  ask  them 
to  respond  orally  and  record  their  responses. 

Step  4:  Orientation 

NOTE  TO  FACILITATOR:  The  Vignette  ox\d  Orientation  materials  in  the 
participant  binders  are  for  their  use  at  any  time.  The  materials  include  tactical 
materials  (e.g.,  OPORD,  maps,  FRA&Os),  project  overview,  FCS/S03  introduction, 
list  of  UAS  capabilities,  agenda,  etc. 

Instructions  to  Participants:  "The  materials  in  your  binder  are  available  for  your  use  as  you 
work  through  the  simulation  vignettes.  Right  now,  we  want  you  to  view  the  multimedia 
Orientation  on  your  workstation  and  ask  questions  as  you  go  along.  ” 

The  workstation  operators  help  each  participant  start  the  Orientation  and  answer  questions  as 
they  arise.  After  the  last  participant  finishes,  the  exercise  director  asks  for  questions. 

Once  all  participants’  questions  have  been  answered,  the  exercise  director  provides  the  following 
directions  for  the  vignettes. 

a.  "You  are  going  to  watch  several  simulation  vignettes  in  which  an  FCS  UAS  could  be 
employed.  As  you  role-play  your  assigned  position,  please  describe  your  reactions, 
thoughts,  and  decisions  as  you  work  to  complete  the  mission  while  you  direct  the 
workstation  operator.  We  are  supplying  devices  to  emulate  tactical  communication.  We 
will  be  audio  recording  your  verbal  responses  and  analyzing  the  content.  It  is  importan  t 
that  you  think  aloud  as  you  work  and  try  to  explain  your  decisions  and  actions 
throughout  the  vignette. 

b.  Feel  free  to  consult  the  materials  in  your  binders  as  needed.  There  is  no  time  limit,  but 
please  move  along  in  order  to  stay  on  schedule. 

c.  Immediately  following  each  vignette,  we  will  conduct  a  brief  review  and  ask  you  to 
review  and  refine  some  TTPs  developed  by  earlier  Soldiers.  You  have  a  set  of  previously 
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d.  developed  TTPs  for  UASs  in  your  binder  materials.  You  will  be  reviewing  and  refining 
these  based  on  your  own  experience  in  the  simulation  vignettes.  Take  a  moment  to  look 
over  these  now  before  we  begin.  ”  (See  Step  1  of  KE  Protocol) 

e.  Do  you  have  any  questions?  (Facilitators  respond  to  questions.) 

Step  5:  Practice  Vignette 

Instructions  to  Participants:  To  familiarize  you  with  the  vignettes,  we  ’re  going  to  conduct  a 
practice  event.  Feel  free  to  ask  questions  as  we  go  along.  As  you  execute  your  assigned  role, 
remember  to  verbalize  your  thoughts  and  actions  so  we  know  what  you  ’re  doing  and  why.  When 
I  ask  a  question,  try  to  answer  it  as  best  you  can.  Please  be  thinking  about  the  TTPs  that  would 
be  appropriate,  even  though  we  won  7  be  developing  TTPs  for  this  practice  vignette. 

NOTE  TO  FACILITATOR:  Let  the  participant(s)  get  a  feel  for  the  simulation  for 
the  f irst  10  minutes,  then  start  asking  questions  from  the  list  below.  Help  them 
understand  the  simulation  environment  and  their  role  in  the  TTP  development 
process.  Record  notes  about  things  you  want  to  explain  at  the  end  of  the  practice 
event. 

1 .  What  are  you  paying  attention  to  in  this  situation?  (observations/cues) 

2.  What  factors  are  you  considering  as  you  plan  a  course  of  action?  (factors) 

3.  What  is  the  best  course  of  action  to  take  at  this  point  and  why?  (key  decision) 

4.  What  is  influencing  the  approach  you  take?  (tactic) 

5.  What  technique/s  are  you  using  at  this  point  to  achieve  the  mission?  (technique) 

6.  Is  there  a  specific  procedure  that  you  would  follow  at  this  point?  (procedure) 

At  the  end  of  the  practice  event : 

•  Facilitator  asks  the  participant(s)  what  questions  they  have  about  the  simulation,  the  vignette, 
their  tactical  role,  or  thinking  aloud. 

•  Facilitator  uses  notes  made  during  the  vignette  to  clarify  or  explain  how  the  participant(s) 
should  be  role-playing  and  thinking  aloud. 

•  Give  the  participant(s)  a  5 -minute  break  while  the  research  team  prepares  for  the  first  TTP 
review/refinement  run. 


C-5 


Step  6:  Think-Aloud  Prompts  for  Vignette  1 


Instructions  to  Participants :  We  will  work  with  TTPs  that  have  been  developed  by  another  group 
of  Soldiers.  Briefly  study  these  TTPs,  as  you  will  be  reviewing  and  refining  them  after  you 
complete  the  simulation  vignette. 

Facilitator's  Action:  Turn  on  audio  recorder!  State  date,  installation,  group 
number,  and  vignette  number. 

Instructions  to  Participants:  We  are  now  ready  to  work  through  the  first  simulation  vignette. 
Your  job  is  to  role-play  in  character  and  verbalize  your  thoughts  and  actions  so  we  can  better 
understand  your  warfighting  process.  I  may  ask  you  a  few  questions  to  help  you  think  aloud.  If 
you  find  it  hard  to  respond  to  a  question  while  you  are  engaged  with  the  simulation,  continue 
working  through  the  vignette  and  answer  at  a  later  time  that  works  best  for  you. 

NOTE  TO  FACILITATOR:  Be  sure  to  cycle  through  each  of  these  questions  as  the 
participant  reaches  key  decision  points.  Record  the  participant's  responses  to  each 
question  on  the  following  page.  While  interactions/dialogue  will  be  audio-recorded, 
having  your  notes  will  speed  the  data  analysis  process.  Make  sure  recorder  is  on! 

1.  Have  the  TTPs  accounted  for  all  of  the  important  factors/cues  associated  with  this 
situation?  (observations/cues) 

2.  Have  the  TTPs  outlined  the  best  course  of  action  to  take  at  this  point  and  why?  (key 
decision) 

3.  What  is  influencing  the  approach  you  take?  (tactic) 

4.  Are  the  technique/s  outlined  in  the  TTPs  enabling  you  to  achieve  the  mission  at  this  point 
in  time?  (technique) 

5.  Are  the  procedures  that  you  are  following  according  to  the  TTPs  at  this  point  effective? 
(procedure) 
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KE  Think  Aloud  Data  Collection  Form 


Vignette  1 

Think  Aloud  Data 

1.  Observations/Cues 

2.  Factors  Considered 

3.  Key  Decision 

4.  Tactic 

5.  Technique 

6.  Procedure 

1.  Observations/Cues 

2.  Factors  Considered 

3.  Key  Decision 

4.  Tactic 

5.  Technique 

6.  Procedure 

1.  Observations/Cues 

2.  Factors  Considered 

3.  Key  Decision 

4.  Tactic 

5.  Technique 

6.  Procedure 

Step  7:  Review/Refinement  of  Previous  TTPs  for  Vignette  1 


Instructions  to  Participants:  “In  your  binders  you  will  find  a  Worksheet  to  use  for  recording 
your  thoughts  about  the  previously  generated  TTPs  for  Vignette  1.  As  you  review  the  set  of 
TTPs,  please  consider  the  actions  you  took  in  the  simulation  vignette  and  determine  how  you 
would  change  or  add  to  this  set  of  TTPs.  After  you  write  down  your  ideas,  we  will  discuss  them 
together.  ” 

INSTRUCTIONS  TO  FACILITATOR:  When  the  participant(s)  finish  their 
worksheet,  discuss  their  ideas  using  the  following  questions. 

1 .  What  are  your  main  thoughts  about  the  TTPs? 

2.  Do  the  TTPs  adequately  address: 

a.  Time  available? 

b.  Troops  available? 

c.  Different  types  of  terrain? 

d.  Presence  and  location  of  enemy? 

e.  Presence  and  location  of  civilians? 

f.  Other  factors  identified  by  the  participant(s)? 

3.  What  are  your  primary  suggestions  for  improving  the  TTPs? 

4.  Given  the  mission  you  just  completed,  what  are  some  negative  outcomes  that  might  result 
from  using  a  UAS?  How  could  these  be  avoided? 

5.  What  capabilities  and/or  attributes  does  the  UAS  need  to  support  mission  accomplishment? 


INSTRUCTIONS  TO  FACILITATOR:  Summarize  the  discussion  and  ask  for  final 
adjustments.  Then  assemble  all  participants  for  a  short  AAR. 

Facilitator's  Action:  Turn  off  audio  recorder. 


C-8 


NOTE:  Materials  for  Steps  8-13  (Vignettes  2-4)  are  identical 
to  those  for  Steps  6-7  (Vignette  1)  and  are  omitted  from  this 
document  for  the  sake  of  brevity. 


You  have  completed  all  vignettes. 


Step  14:  Summary  Review 

INSTRUCTIONS  TO  FACILITATOR:  This  stage  follows  completion  of  the  last 
vignette.  Please  assemble  all  participants  into  one  group  and  discuss  the  following 
questions. 

1 .  How  will  the  use  of  a  UAS  vary  by: 


a.  Different  types  of  missions? 

b.  Time  available? 

c.  Different  types  of  terrain? 

d.  Troops  available? 

e.  Presence  and  location  of  enemy? 

f.  Presence  and  location  of  civilians? 

2.  What  further  thoughts  do  you  have  about  attributes/components  needed  by  the  UAS  to  fully 
support  mission  accomplishment? 

3.  Is  there  a  specific  order  in  which  areas  should  be  observed  by  the  UAS?  If  so,  what  should 
this  order  be? 

4.  What  factors  play  into  or  influence  the  surveillance  sequence? 

5.  What  type  of  target  location  method(s)  would  be  optimally  suited  for  a  UAS?  (Examples 
include  grid  coordinates,  polar  coordinates,  shift  from  known  point,  or  other.) 

6.  Would  there  be  a  difference  in  how  a  UAS  would  be  used  by  a  platoon  versus  a  company? 
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7.  What  are  the  key  concepts  a  leader  must  be  aware  of  to  optimally  use  the  UAS? 

8.  If  you  were  provided  with  a  UAS,  in  what  situations  would  you  use  it?  In  what  situations 
would  you  hesitate  to  use  it  or  even  avoid  using  it? 

9.  Consider  the  following  procedural  aspects  of  UAS  employment: 

a.  What  conditions  detennine  whether  to  use  a  UAS  or  not? 

b.  How  many  do  you  fly;  how  high  does  it  fly;  how  fast  does  it  move;  what  flight  pattern 
do  you  program;  and  who  controls  the  UAS  during  execution  of  a  mission? 

c.  Where  should  the  UASs  be  located  in  relation  to  ground  forces? 

10.  What  else  would  you  consider  in  using  a  UAS?  Why? 

1 1 .  Please  briefly  review  your  revised  TTPs  for  each  vignette.  Would  you  make  further 
changes?  If  yes,  what  would  you  change? 

12.  Do  you  have  any  comments  on  the  TTP  development  method  or  this  session  in  general? 

Step  15:  Participant  Feedback  Questionnaire 

Instructions  to  Participants:  “If  you  would  now  turn  to  the  Participant  Feedback  questionnaire, 
please  fill  out  the  questionnaire.  It  should  take  only  a  few  minutes  but  we  would  really 
appreciate  it  if  you  would  take  your  time  and  provide  us  with  thoughtful  responses.  ” 

Step  16.  Wrap  Up 

•  Facilitators  collect  all  materials  from  participants. 

•  Exercise  director  answers  final  questions  and  explains  research  payoff. 

•  Exercise  director  thanks  participants  and  releases  them. 

•  KE  facilitators  download  digital  audio  files  to  computer  and  turn  off  recorders. 

•  Facilitators  label  and  file  all  paperwork  in  appropriate  folders. 


End  of  Session 
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Review/Refinement  Worksheet  (Participant) 


NOTE  TO  FACILITATOR:  This  worksheet  is  for  use  with  participants  who  are 
conducting  Single-Source  or  Multi-Source  TTP  review/refinement  ONLY.  If  you 
are  working  with  a  group  that  is  doing  TTP  Development  please  use  the  alternate 
data  collection  package  found  in  Development  Data  Collection  Package. 

Mission  1:  Cordon  &  Search 

Original  TTP  Summary: 


Following  the  simulation,  please  specify  if  the  following  TTP  should  be: 

K-kept  as  is 
M-modified 
D-deleted 

If  you  want  to  Keep  the  item  as  is,  specify  why.  What  factors  make  it  good? 

If  you  want  to  Modify  the  item,  just  below  that  item  explain  why  it  needs  to  be  changed.  How 
does  changing  it  improve  it? 

If  you  want  to  Delete  an  item,  specify  the  shortcomings  of  that  item. 


1.  Mission: 

a.  Tactic: 


Please  circle  one 
K  M  D 


b.  Technique: _  K  M  D 

c.  Procedure:  K  M  D 


d.  Consideration: 


K  M  D 


2.  Enemy: 

a.  Tactic: _  K  M  D 

b.  Technique: _  K  M  D 
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K  M  D 


c.  Procedure: _ 

d.  Consideration: 

3.  Terrain: 

a.  Tactic: _ 

b.  Technique: _ 

c.  Procedure: _ 

d.  Consideration: 

4.  Troops: 

a.  Tactic: _ 

b.  Technique: _ 

c.  Procedure: _ 

d.  Consideration: 

5.  Time: 

a.  Tactic: _ 

b.  Technique: _ 

c.  Procedure: _ 

d.  Consideration: 

6.  Civilians: 

a.  Tactic: _ 

b.  Technique: _ 


K  M  D 

K  M  D 

K  M  D 

K  M  D 

K  M  D 

K  M  D 

K  M  D 

K  M  D 

K  M  D 

K  M  D 

K  M  D 

K  M  D 

K  M  D 

K  M  D 

K  M  D 
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c.  Procedure: _  K  M  D 

d.  Consideration:  K  M  D 


NOTE:  Participant  worksheets  for  Vignettes  2-4  are  identical 
to  those  for  Vignette  1  and  are  omitted  from  this  document 
for  the  sake  of  brevity. 
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Appendix  D 

Flex  TTP  Participant  Feedback  Questionnaire 


Date _  Participant# _  Group# _  Role:  Cdr/PL  Development / Yetting 


Instructions:  The  questions  below  ask  for  your  opinions  about  the  materials  and  tools  you  used 
today  to  develop  future-focused  TTPs.  Write-in  comments,  both  positive  and  negative,  are 
encouraged.  Please  use  a  separate  sheet  of  paper  if  you  need  additional  space. 


1.  How  much  do  vou  agree  or  disagree  that  the  Orientation: 

Circle  One  for  Each  Item 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

a.  Set  the  stage  well  for  the  session? 

1 

2 

3 

4 

5 

b.  Provided  everything  1  needed  to  know  about  the  method? 

1 

2 

3 

4 

5 

c.  Contained  accurate  information  about  FCS  Spin  Out  3  capabilities? 

1 

2 

3 

4 

5 

d.  Was  clearly  presented  and  easy  to  understand? 

1 

2 

3 

4 

5 

Comments  and  Suggestions: 

2.  How  much  do  vou  agree  or  disagree  that  the  Vignettes: 

Circle  One  for  Each  Item 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

a.  Set  realistic  conditions  for  thinking  thru  UAS  employment? 

1 

2 

3 

4 

5 

b.  Caused  me  to  consider  and  decide  among  courses  of  action? 

1 

2 

3 

4 

5 

c.  Covered  a  reasonable  mix  of  missions? 

1 

2 

3 

4 

5 

d.  Portrayed  realistic  enemy  organizations  and  doctrine/tactics? 

1 

2 

3 

4 

5 

e.  Provided  sound  tactical  materials  (road  to  war,  OPORD,  FRAGOs)? 

1 

2 

3 

4 

5 

f.  Were  clearly  presented  and  easy  to  understand? 

1 

2 

3 

4 

5 

Comments  and  Suggestions: 

3.  How  much  do  vou  agree  or  disagree  that  the  Simulation: 

Circle  One  for  Each  Item 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

a.  Portrayed  the  tactical  environment  with  sufficient  realism? 

1 

2 

3 

4 

5 

b.  Demonstrated  the  UAS  capabilities  adequately? 

1 

2 

3 

4 

5 

c.  Represented  enemy  elements  and  capabilities  realistically? 

1 

2 

3 

4 

5 

d.  Gave  me  enough  control  and  flexibility  of  my  unit’s  behaviors? 

1 

2 

3 

4 

5 

e.  Enabled  me  to  visualize  the  battlefield  well? 

1 

2 

3 

4 

5 

f.  Adequately  supported  tactical  communications? 

1 

2 

3 

4 

5 

Comments  and  Suggestions: 
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4.  How  much  do  vou  agree  or  disagree  that  the  Schedule: 

Circle  One  for  Each  Item 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

a.  Took  too  long  for  the  TTPs  we  developed? 

1 

2 

3 

4 

5 

b.  Asked  me  to  do  too  much  in  the  time  1  had  available? 

1 

2 

3 

4 

5 

c.  Allocated  the  right  amount  of  time  for  the  various  activities? 

1 

2 

3 

4 

5 

d.  Enabled  me  to  spend  my  time  efficiently? 

1 

2 

3 

4 

5 

e.  Organized  my  activities  in  the  right  sequence? 

1 

2 

3 

4 

5 

f.  Gave  me  enough  break  time  when  1  needed  it? 

1 

2 

3 

4 

5 

Comments  and  Suggestions: 

5.  How  much  do  vou  agree  or  disagree  that  the  Procedures: 

Circle  One  for  Each  Item 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

a.  Facilitated  development  of  sound  TTPs? 

1 

2 

3 

4 

5 

b.  Encouraged  me  to  explore  all  aspects  of  the  TTPs? 

1 

2 

3 

4 

5 

c.  Achieved  a  good  balance  between  role-playing  and  thinking  aloud? 

1 

2 

3 

4 

5 

d.  Enabled  me  to  work  effectively  with  the  right  focus? 

1 

2 

3 

4 

5 

e.  Captured  my  thoughts  and  insights  accurately? 

1 

2 

3 

4 

5 

f.  Gave  me  a  chance  to  fully  express  myself? 

1 

2 

3 

4 

5 

g.  Took  advantage  of  group  interaction  and  collaboration? 

1 

2 

3 

4 

5 

Comments  and  Suggestions: 

6.  What  are  your  general  impressions  of  the  TTP  development  method  you  used  today? 


7.  How  would  you  improve  the  TTP  development  method  you  used  today? 
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